Unstructured Data Ingestion

Ask the Community
,
1

We are ingesting unstructured data from S3 AWS, but the problem here is file formats are undefined. The S3 bucket folder contains 18 different file formats as of now and in future there might be a different file format coming in. In our POC, we have imported all the files manually and converted(gotenberg library) them into PDFs so that they can processed properly using LLM.

Now I’m confused of how to ingestion from S3. batch sync or media set sync. And if it is media set sync then is multimodal sync the best option? Ultimately I had convert them into PDFs and then use in the LLM pipeline. Any recommendations on this will be very helpful.

Thank you

2

Hi Vardhan,

I generally prefer to pull them into an unstructured dataset, before moving them to the mediaset. This gives me the flexibility to apply custom logic (e.g. native extraction) and catch errors in media conversion (formatting errors, locked files, etc.) without having to rerun the pipeline.

From there, I would first work on the files that the mediaset supports out of the box. The multimodal mediasets have have some limitations, so you need to figure out whether these will end up being blockers for your proposed workflow.

If they are blockers, you can create custom transforms for each unsupported file format, not supported by the mediaset schema. It’s relatively easy to do, and since they are reusable, their value will compound over time.

3

Thank you Jake,

Extension to my earlier question.

We have an S3 source that delivers a full snapshot of unstructured files on every sync — not deltas. These files feed into an LLM vision analysis pipeline downstream, which is expensive (multiple LLM calls per file). Re-processing the entire set every build is costly and slow, so we’d like to process only new or changed files. The unstructured data is also linked to structured data in the Ontology through a join table.

What is the recommended pattern in Foundry for making unstructured data processing incremental when the source always sends a full load? Is there a way to detect changes to existing files (same path, different content) between sync transactions.

Thanks,
Vardhan

4

I am unsure about what you mean by the source always sending a full load. Are you not pulling this into Foundry, or is another service pushing this into Foundry?

If you’re in charge of the ingestion, you have several ways to mitigate against that:

  • the Data Connector you use to import the data to Foundry can work in Incremental-mode
  • the Pipeline you use to transform the data can build in Incremental-mode

Alternatively, I would make sure to have a step in your pipeline, that deduplicates previous files from processing. Incremental builds will generally solve for this, but if it for some reason does not, a file hash can also be used to exclude already processed files.

In that scenario, your only “wasteful” step would be the reingestion of all the files in the initial step. After that you can process only the files that do not already exist.

That said, if files are currently being pushed to Foundry, I would look into whether you could change this to “pull” and let Foundry do the ingestion, as this is much cleaner and more predictable.

5

Thanks for the response. To clarify — we are pulling from S3 using a Foundry connector. The issue is that the upstream system refreshes/re-uploads all files in the S3 bucket on every cycle, so even though Foundry does the pulling, the source files all get new timestamps and the connector sees everything as “new.”

What we’ve done for Structured Data:

  • The S3 connector is set up as a snapshot sync (since the source always sends the full set)
  • In the downstream transform, we use @incremental with snapshot_inputs for the source dataset
  • We compare the current snapshot against the previous output using primary key + last_modified timestamp to identify new, modified, and deleted records
  • Only the changed records are appended to the output
  • We haven’t deduplicated the final output dataset — the Ontology handles duplicate PKs across transactions via “latest transaction wins”

What we’re planning for Unstructured Data (LLM vision pipeline):

  • Accept the full ingestion cost from S3 into a Foundry media set (this is just file transfer, relatively cheap)
  • In the downstream transform, use content hashing (SHA-256) to detect which files are truly new or changed vs. just re-uploaded with the same content
  • Maintain a processed-file registry that tracks file paths and their content hashes
  • Only send genuinely new/changed files through the expensive LLM pipeline
  • This way the costly LLM processing stays incremental even though the source always delivers a full snapshot

Does this approach sound right to you, or would you suggest a different pattern?

Thanks,
Vardhan