This is a newbie question and seems like it should be simple, but I am struggling to find a way to have my Python code automatically update when a dataset is updated from a Python.
Looking at my visual below, I have a couple Pipelines that are running fine and updating the datasets as new data arrives. However, I am not able to get the Python code to automatically run to do the additional transforms needed and update the data sets. Is there a way to call the Python code from the Pipeline? Or do I need to configure the scheduler in a way to have the Python code run?
Hi @carnold, I think what you are looking for are schedules. If you want your pipeline builder/code repo to run after certain input datasets have been updated, you can set a schedule on the outputs created by the pipeline builder/code repos.
In pipeline builder, you can right click on a target dataset, and clicking Actions → Manage Schedules. This should take you to see any existing schedules and where you can also create a new schedule. When creating a new one, if you want your input updating to be a trigger, select from the menu “When specific resource(s) update” and mark the input datasets as triggers. Hope this helps!
