Using a pipeline output in its input

[mrestuccia]

Hey team,

When building pipelines, I often find the need to use a pipeline’s output as an input as well. Something like:

Inputs
Dataset B, Dataset A (Last outputted Version) → Some Transformation → Dataset A

Via code repos, you can do this by writing incremental transforms. I am wondering if there is a recommended way to do something similar in Pipeline Builder.

Thank you!

[helenq]

Hi @mrestuccia we’re currently working on being able to output checkpoints as materialized outputs. This would only output the most recent transaction (so for an incremental output this would only show the latest rows that were appended, but for a snapshot this would be the entire dataset) Would this meet your needs here or could you tell us more about your usecase?

[mrestuccia]

Thank you @helenq, this is exactly what I am looking for!