Differences and Usage of Expectations in Pipeline Builder vs. Build Status Checks in Dataset Preview

Please explain the differences and best practices for using expectations added from Pipeline Builder, as documented here:

  • https://www.palantir.com/docs/foundry/pipeline-builder/dataexpectations-overview

and build status checks added from Dataset Preview, as documented here:

  • https://www.palantir.com/docs/foundry/pipeline-builder/dataexpectations-configure-health-check.

The types of expectations available from Pipeline Builder appear to be more limited. However, both can be viewed in the health checks.

Expectations should be used to make assertions about your data and/or how your pipeline is functioning. Health checks are better suited for monitoring and alerting. Note that expectations are run synchronously with a build while health checks are evaluated async (after the build finishes).

Much of this functionality can be replicated using health checks but importantly expectations allow you to fail the build and not commit a new transaction to the output dataset. This is really important if you have business critical decisions (or automated processes) being made on the result of a pipeline.

Consider this example:

  • You are calculating the number of days an invoice might be overdue in the pipeline
  • The source system has an edge case where it is not able to parse a due date from the invoice, but stores this due date as an unsigned int, meaning a 0 indicates 1970-01-01.
  • Your pipeline will then calculate this invoice as being many thousands of days overdue.
  • This could trigger an automation to send an email to the customer indicating that their future orders will be cancelled.

If you had an expectation that the number of days overdue should be <=365 then you would be able to fail the build without committing the erroneous data, which would give you time to manually inspect the pipeline and create a fix for this bug.

4 Likes

Thank you for your clear explanation of the differences and the helpful example. It has greatly improved my understanding.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.