# Pipeline builder incremental mediasets pdf extraction **URL:** \ **Category:** Ask the Community\ **Tags:** pipeline-builder, media-sets\ **Created:** [September 13, 2024, 8:53pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279 "2024-09-13T20:53:30Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jkishk](https://avatars.discourse-cdn.com/v4/letter/j/ac91a4/32.png) [@jkishk](https://community.palantir.com/u/jkishk)\ **Post date:** [September 13, 2024, 8:53pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/1 "2024-09-13T20:53:30Z")
Hi, Is incremental computation mode supported for media sets? I am not seeing the option under the media set icon in pipeline builder. My workflow is pdf to text with OCR and then text to language model. To avoid wasting tokens on documents that have already been processed I have created an intermediary dataset that holds the extracted text, written with append rows. Because OCR is not guaranteed to be deterministic, I need to key on file name. This works, but still involves extra OCR extractions on documents that have already been processed. Also worth noting I am unable to extract the text from a media reference from a dataset due to permission issues. ---
**Author:** ![helenq](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@helenq](https://community.palantir.com/u/helenq)\ **Post date:** [September 13, 2024, 10:33pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/2 "2024-09-13T22:33:30Z")
Currently incremental media sets are not supported in Pipeline Builder (https://community.palantir.com/t/does-pipeline-builder-support-incremental-builds-off-of-a-mediaset/182) If you’re using the use llm node you can also skip recomputing rows which will help you save time and costs: https://www.palantir.com/docs/foundry/pipeline-builder/pipeline-builder-llm/#skip-computing-already-processed-rows GPT4o in Pipeline Builder also supports vision and will roll out next week! ---
**Author:** ![HugoRodrigues](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/hugorodrigues/32/999_2.png) [@HugoRodrigues](https://community.palantir.com/u/HugoRodrigues)\ **Post date:** [October 9, 2024, 4:49am UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/3 "2024-10-09T04:49:29Z")
Hi @jkishk Is it required to process the mediasets in pipeline builder? To complement @helenq 's answer - You could use @incremental(v2\_semantics=True) or @incremental (semantic\_v2) in code repositories. Yet, it is still beta but it works. ---
**Author:** ![jkishk](https://avatars.discourse-cdn.com/v4/letter/j/ac91a4/32.png) [@jkishk](https://community.palantir.com/u/jkishk)\ **Post date:** [October 9, 2024, 2:20pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/4 "2024-10-09T14:20:25Z")
Hi @Hugo, the transform does not have to be in Pipeline Builder. As we put the workflow into prod, we may have to move the transform into code repositories. ---
**Author:** ![CodeStrap](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/codestrap/32/138_2.png) [@CodeStrap](https://community.palantir.com/u/CodeStrap)\ **Post date:** [October 9, 2024, 3:00pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/5 "2024-10-09T15:00:15Z")
I’ve been setting up my own incremental logic to subtract rows based on a custom join. I use document id and page number, but you could optionally check the extracted text to see if it’s changed. Full credit to the Palantir team who showed me this approach and provided the boilerplate: ```auto def get_incremental_data(ctx, input_dataset, output_dataset, limit_rows=True, limit=2): # We enforce the read of the input dataframe as a snapshot, via the snapshot_input decorator input_df_all_dataframe = input_dataset.dataframe(mode="current") if hasattr(ctx, '_is_incremental') and ctx._is_incremental: # We read the current output to see what we already processed in previous builds # Note: We have to specify the schema for the first run # page_id is a hash of the page number and media set item rid which should be unique for each page and stable between builds out_schema = T.StructType([ T.StructField('page_id', T.StringType()), # T.StructField('page_number', T.IntegerType()), ]) output_df_previous_dataframe = output_dataset.dataframe('current', out_schema) # ==== Example processing here ==== # We diff the input with the current output, to find the "new rows". # We do this with a LEFT ANTI join : A - B <==> A LEFT ANTI B KEY = ["page_id"] new_rows_df = input_df_all_dataframe.join(output_df_previous_dataframe, how="left_anti", on=KEY) else: # On first run new_rows_df = input_df_all_dataframe # We had a timestamp for easier tracking/debugging/understanding of the example new_rows_df = new_rows_df.withColumn('incremental_ts', F.current_timestamp()) # 2. Conditional row limiting based on the parameter if limit_rows: new_rows_df = new_rows_df.limit(limit) return new_rows_df ``` Like you said though OCR isn’t deterministic. But if you are lucky enough to be extracting the text layer you cold know with a fair degree of certainty if the text of the page has changed in any way. If you are processing images, audio, etc you might be able to use checksums. Even with incremental support for media sets it might not have the level of granularity your want. ---
**Author:** ![HugoRodrigues](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/hugorodrigues/32/999_2.png) [@HugoRodrigues](https://community.palantir.com/u/HugoRodrigues)\ **Post date:** [October 10, 2024, 6:56am UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/6 "2024-10-10T06:56:47Z")
Hi @jkishk We have been using v2\_semantics and haven’t had issues (see code below). ```auto from transforms.api import transform, incremental from transforms.mediasets import MediaSetInput, MediaSetOutput @incremental(v2_semantics=True) @transform( output=Output("..."), media_refs=Input("..."), mediaset=MediaSetInput("...") ), ) def compute(ctx, output, media_refs, mediaset): def extract_text_from_pdf(media_item_rid, page_number): response = mediaset.transform_document_to_text_raw(media_item_rid, page_number) return response.read().decode("utf-8") text_udf = F.udf(extract_text_from_pdf, T.StringType()) df = media_refs.dataframe().withColumn('extracted_text', text_udf(F.col('mediaItemRid'), F.col("page_number"))) output.write_dataframe(df) ``` I am not sure how close to GA v2\_semantic is but maybe your colleagues closer to the development team can elaborate further. ---
**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\ **Post date:** [March 14, 2025, 6:21am UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/7 "2025-03-14T06:21:13Z")
---
**Author:** ![Amarkshi143](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@Amarkshi143](https://community.palantir.com/u/Amarkshi143)\ **Post date:** [April 28, 2025, 3:40pm UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/8 "2025-04-28T15:40:36Z")
Hi Rodrigues, I am trying to apply incremental operator using v2\_semantics on mediaset input of pdfs. I am getting the **“object does not have an attribute `list\_media\_items\_by\_path\_with\_media\_reference”** When I remove the incremental decorator then I am able to preview the data. Have you encoutered the same issue? **code snippet:** @incremental(v2\_semantics=True) @transform( output=Output(“..”), mediaset=MediaSetInput(“..”) ) def compute(ctx, output, mediaset): # List all media items in the media set media\_items\_listing = mediaset.list\_media\_items\_by\_path\_with\_media\_reference(ctx) ---
**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\ **Post date:** [January 28, 2026, 11:40am UTC](https://community.palantir.com/t/pipeline-builder-incremental-mediasets-pdf-extraction/1279/9 "2026-01-28T11:40:06Z")