This is a duplicate of this issue which does not have an accepted answer. In many cases notifying a group when a job that effects backing data finishes is used as a trigger for workflows. I’m not sure this is the best pattern but it’s one we get a lot. There are also times when external systems need to notified but this can usually be achieved using external pipelines. That is likely the solution we will use.
Automate does not appear to have a viable option for build notifications:
Run on all objects: Periodically runs effects on all objects in a given object set.
Metric changed: Triggers when an aggregated metric increases or decreases.
Threshold crossed: Triggers when a metric threshold is crossed or when a function returns true.
None of these options === build completed. Also it would require the dataset back an ontology object.
Health checks have time since last build but that won’t trigger when builds are successful. Also using health checks for this seems like an anti pattern as it would increase noise as these are not alerts just notifications.
One option would be to create a multi-output transform with the new output holding metrics about the latest state, i.e. uuid & timestamp, that builds incrementally and is materialized as an Object type. Then, an Automate listens for new objects added to set - representing a successful build - and the notification can trigger.
Actual dataset - This one with your actual business logic
Incremental - This will always store the latest build timestamp of dataset. For that we can hit the API - List Transactions Of Dataset • API Reference • Palantir that gives transaction details of dataset. we can do the comparison with latest timestamp from API output with timestamp from incremental (in ideal case timestamp should not match and should be greater than the timestamp from incremental dataset).
This will allow us to trigger automation in 2 ways.
Objects added to set: Triggers when a new object appears in the set (set pk as uuid, to create a new record every time incremental sync to dataset)
Threshold crossed: Triggers when a metric threshold is crossed or when a function returns true. (Create a flag column in incremental dataset, if latest timestamp from transaction is greater than timestamp stored in incremental dataset)
Hey thanks for replying. We were looking for a way to do this without introducing additional datasets. Ideally this would have first class support in build configs. So hopefully Palantir will add it. What we opted for was to just trigger notifications form the transform itself using some custom code. But your solutions would probably work as well. I did not test it though.
This is a long standing feature request. The original answer I got was, why not use an export task as child of your transform from where you can trigger the external process.
By now export tasks have been deprecated for 2 years
streaming logs of a namespace or project. However, you would have to handle and filter the log stream and I am also not sure if there are events for build finished.
