Hi,
Is there a way for me to create a health check that checks for the freshness of a specific value? I would like to be able to alert when an incremental dataset hasn’t seen a new row appended with a specific value in a column within a specified time frame.
Thanks!
Hello. The Data Freshness health check takes a time column in your data that represents the creation time of the row. This will check that the maximum value in this column is no more than a configured amount greater than the time of the most recent transaction.
https://www.palantir.com/docs/foundry/data-health/checks-reference/#data-freshness
For more granular control, you could create a temporary column containing the current Timestamp, then use Data Expectations to compare the value of your target column against the temporary column:
https://www.palantir.com/docs/foundry/transforms-python/data-expectations-reference/
Gotcha, thanks! That makes sense. Do you think we could use a conditional check (https://www.palantir.com/docs/foundry/transforms-python/data-expectations-reference#conditional) here too?
Suppose we had two columns, one containing a piece of data and another with a timestamp. I’d like to be able to monitor that there exists at least one row in the last X minutes/hours/days (using the timestamp column) where the data column’s value matches a specific pattern.