If I have a pipeline that outputs a dataset using the Snapshot replace write mode, how can I force a reset (i.e. so that the next build will do a regular snapshot)?
Hi rwaldman,
Not sure I understand the question - do you have a quick code example? If you use replace as a mode in a pipeline with an incremental pipeline, it will trigger a snapshot of said output with whatever you write in this new transaction. This is often coupled with a “previous” mode before, so if you want to “start fresh” you can add logic in your pipeline to empty out this “previous” loaded dataset if you want this to happen automatically on certain conditions. You can also bump the semantic version if it’s a one-off and should be done manually.
To be clear, this is in pipeline builder, not a code transform.
The pipeline is essentially set up with a snapshot input, and a snapshot-replace output, so that we maintain records from the input that have been deleted, and update any records that have new values.
So I’m looking for something similar to a semantic version bump / pipeline replay, where the snapshot-replace output of the pipeline will start a fresh snapshot without any of the historical data.
I see. The easiest way would probably then to temporarily change the write mode to default, deploy, then re-set it to snapshot replace.