Pipeline Builder Changelog Timestamp Columns

Ask the Community
1

When using the “Changelog” write mode in pipeline builder, two columns are added to the output dataset - timestamp and is_deleted.

Which timestamp does the timestamp column represent? Is it the first time that row was seen? Is it the most recent time that row was seen? Is it the time it was deleted?

2

Is this for Object Storage v1?

  • The timestamp column represents the most recent time that the row was seen. This column is used to track changes to the data, and it reflects the time at which the data was processed in the pipeline.
  • The is_deleted column indicates whether the row has been deleted

https://www.palantir.com/docs/foundry/data-integration/change-data-capture#change-data-capture-in-pipeline-builder

https://www.palantir.com/docs/foundry/pipeline-builder/outputs-add-dataset-output/#configure-write-mode

3

My __changelog_timestamp column is recorded in Integer, how can I change this to datetime?

4

Hi @jyoo, having the __changelog_timestamp column available a timestamp/datetime also is in the works :grinning:, and it should be available soon. I will update in this thread when the column has been added.