Building incrementing dataset from snapshot dataset in Pipeline Builder

kevcam4891 · December 30, 2025, 5:28pm

Is it possible to build INCREMENT output from a SNAPSHOT input in Pipeline Builder like you can in Code Repositories? Example in CR:

@incremental(snapshot_inputs=['input_dataset'], semantic_version=2)
@transform.using(
    output_dataset=Output("ri.foundry.main.dataset.74c881f9-6684-4799-8fb5-0da0e734f588"),
    input_dataset=Input("ri.foundry.main.dataset.f6d25fd0-55a5-4db5-80ab-700b541ecfc0"),
)
def compute(input_dataset: LightweightInput, output_dataset: LightweightOutput) -> None:
    df = input_dataset.polars()
    df = df.with_columns(pl.lit(datetime.now(timezone.utc).isoformat()).alias("foo"))
    output_dataset.write_table(df);

I saw this dropdown in PB:

…but it doesn’t seem to work because the input DS is of type snapshot (it’s built by Contour).

This code above is a great recipe for building a historical reporting dataset, and I’m appreciative of it, but I’d like to give this power to no-code folks on the team.

Thanks for the help!

helenq · January 5, 2026, 2:06pm

Hey! Depending on what you’re trying to do you could use the write modes to always append the rows from a snapshot input: