Hello again!
What is the best practice for handling runtime errors in a pipeline of multiple interlinked transforms (5+), managed in a code repository?
I want to log any runtime exceptions into a centralized dataset. Do I need to create a separate error dataset for each transform, or is there a way to use a single shared error dataset across all transforms?
How can I design a centralized exception-handling mechanism for this type of pipeline?
If you go for an approach of trying to catch exceptions the builds will report as successful which will make a number of other parts of the platform a fair bit less useful. Eg downstream schedules builds will still kick off, data health will report success etc.
Why do you want to try to catch exceptions centrally?
Why not have health checks/monitoring views and then allow the pipeline to fail when there is an issue? I’d recommend data expectations to make sure your pipelines don’t encounter any data they’re not expecting.
Thank you!
So I have to trigger an external service whenever there is a run time exception and send the the exception as a payload. I wanted to centralize it so that I do not have to call this external service on every transform.
Thanks again!
Ah that makes sense, thanks for the additional info!
I’m imagining the transform is likely a connection with the external system - is this something you’ll do multiple times in a transform? Eg you’ll do 1000 calls, and 2 of them might fail? In that case I agree it’s maybe not worth failing the entire transform.
If you want to put the errors into a dataset you’ll need separate ones per transform (datasets can only be ‘outputted’ to by a single source) but could use a view to union all of these errors afterwards. Alternatively you could look at using OSDK to create objects directly from the transform - this would have the advantage of them being created ‘live’ (before the xform is completed).
You could look at making this ‘error logging’ functionality into a library which you can import across the stack.