I created a pipeline for text extraction from pdfs.
Now when I upload new pdf’s to the media set over time, how do I make the the pipeline recognise the changes and dynamically update the knowledge tree.
To keep your outputs up-to-date it’s a good idea to configure your pipeline for incremental processing in a Code Repository. This means that instead of reprocessing everything every time, it will only focus on new or changed files. This approach saves time and resources. In code repositories, you can create transforms that specifically identify and process just the new or modified files, making the whole process much more efficient.
Please check this - www.palantir.com/docs/foundry/media-sets-advanced-formats/transforming-media#transforming-media
Update: I’ve implemented the incremental Code Repository transform successfully. New PDFs are being detected and copied to PDF Processed incrementally. Chunks and Use LLM are processing new content correctly. However my final output datasets (Chunks copy, Entities copy, Join table copy) in Pipeline Builder are not automatically rebuilding after the incremental run. The outputs are in the same job group and deployed. Schedule is set to trigger when PDF Processed updates. Any advice on getting the final Pipeline Builder outputs to cascade automatically?
Thank you for your help.
