Summary
Pipeline Builder supports configuring data expectations (Primary Key, Row Count, Value is not null, etc.) through the UI, but these are not exposed in the DSL. I’d like to request that expectations be added as a first-class concept in the Pipeline Builder DSL so they can be managed programmatically.
Motivation
When building production-grade pipelines, data expectations are essential guardrails. Currently, if you author or modify a Pipeline Builder pipeline via the DSL (e.g., through AIP / AI-assisted development, or programmatic pipeline management), there is no way to define expectations in code. They must be manually configured through the UI after the fact.
This creates several problems:
-
Expectations are invisible in the DSL — When reading a pipeline’s definition via
get_pipeline_builder_definition, expectations are not included. There’s no way to audit or review them programmatically. -
AI-assisted pipeline development can’t add expectations — Tools like AIP that generate Pipeline Builder DSL code cannot configure expectations, leaving a critical gap in automated pipeline creation.
-
No version control — Unlike the rest of the pipeline definition (transforms, expressions, outputs), expectations are not tracked in the DSL code, making it harder to review changes or roll back.
-
Manual step breaks automation — In a workflow where pipelines are created and managed programmatically, requiring a manual UI step for expectations breaks the automation chain.