I have a production pipeline and the environment is locked. Whenever the upstream data source is updated, the schedule triggers the build. The build succeeds, but the transaction is aborted, i.e, no data is written to the dataset.
When I unlocked the env and then trigger build, transaction is successful and data is written.
How do I fix this issue? I want production environment to be locked.
Is this an incremental or snapshot (or streaming) pipeline? Are the transactions aborted because there are no new data to add to your output? Also confirming that no logic is changing it’s just the input dataset that is updating?
This is incremental pipeline. There is data to update but the data isn’t updated because env is locked. When I unlock the env and retrigger the build, the data is updated. No logic change.
Hey @manangoel sorry for the delay! It’s a bit hard for me to debug what’s going on with the current information so some follow up questions:
If you trigger the build off of the dataset instead of the schedule (outside of pipeline builder) does it succeed? Or does it also abort
Are you able to share a screenshot of the schedule that you have configured (the side panel)? Wondering if you have anything checked off in the advanced options
If possible would be helpful to see the build history of the output in question as well