We are ingesting unstructured data from S3 AWS, but the problem here is file formats are undefined. The S3 bucket folder contains 18 different file formats as of now and in future there might be a different file format coming in. In our POC, we have imported all the files manually and converted(gotenberg library) them into PDFs so that they can processed properly using LLM.
Now I’m confused of how to ingestion from S3. batch sync or media set sync. And if it is media set sync then is multimodal sync the best option? Ultimately I had convert them into PDFs and then use in the LLM pipeline. Any recommendations on this will be very helpful.
Thank you