Expected behaviour of a schedule with a trashed dataset

Ask the Community
1

Hi!

I am aware that schedules now stop running if they contain a dataset which is trashed. Does this mean, that if a dataset is trashed at any point, there is a risk the schedule silently stops running with no failures / alerts?

If so: we should ideally have a monitoring view health check which allows me to check for a “Time Since Last Updated”, at the schedule level (rather than the individual dataset level).

Thanks for your help!

2

It’s correct that a schedule will fail to run if any of the datasets that it’s building or using as input is trashed. The failure will be visible in the history tab of the schedule metrics page [1].

These failures shouldn’t be silent given that you have a health check or monitor rule:

  • If you have an existing Schedule status health check configured on the schedule, then this check should fire and alert the subscribed users [2].
  • Similarly, if you have a Consecutive schedule failures monitor rule (with the configured failure threshold being 1), that rule is also expected to fire [3].

[1] https://www.palantir.com/docs/foundry/building-pipelines/view-modify-schedules/#view-metrics
[2] https://www.palantir.com/docs/foundry/learning-data-dataeng-06/10/
[3] https://www.palantir.com/docs/foundry/maintaining-pipelines/monitoring-views-intro/#create-monitoring-rules

3

Thanks @btuluce ! This clears things up.