Adding two flags: is_created and is_updated by comparing the current build of a dataset with its previous build using a primary key. Is this possible? If yes, how can be done?
There is different ways you can do that - what are you trying to achieve?
The most common way is to have have a seperate timestamp column last_updated_ts.
If this is for a one off analysis, you can use Contour and select an older transaction when selecting the dataset.
Then you can do an anti join with the newer transaction to get the remaining rows.
Have you checked out incremental transforms?
https://www.palantir.com/docs/foundry/transforms-python/incremental-overview/
With different read and write modes you can identify which rows have been added/modified/deleted since the last time the transform ran.
1 Like