How can I start incremental pipeline from a specific transaction

Ask the Community
1

Hello, I have a very large historical dataset. I want to apply some transformations and create a new dataset that’s derived from the first dataset. However, since the dataset is really large, I want to only consider the data from a specific date and make it incremental. I know how to make it incremental starting from today but I want to make it incremental starting from June 1st.

Is there a way to make an incremental build starting from a specific transaction of the input dataset? Alternatively, is there another way I can achieve what I want?

2

If you have a column in your dataset that has the timestamp can you filter your output to only keep the data from June 1st onwards and then make all future transactions starting from today incremental?

What would be the benefit of making a dataset incremental starting from a past date since you already computed those rows in your snapshot transaction?