I did setup a webhook to a particular API endpoint to fetch some data. The webhook is only pulling data.
Now I want to automate this, to run every 5 minutes to pull this data and save it in a dataset.
What is the most straighforward way to do so ?
Can I just call this webhook in an external transforms so I don’t have to code again the API calls in python ?
Can I just schedule the webhook somehow ?
The intent is to store this data to have an history of what a particular endpoint is exposing, hence objects are not necessary for it.
To achieve this you could wrap the webhook in a typescript function, create an action that calls that function, and then use Automate to schedule that action. In that function, you can create result objects for the results of each webhook execution.
Alternatively like you mentioned you could use external transforms. Unfortunately there’s no first class way to call webhooks from external transforms. One workaround would be to:
Create an action that calls the webhook
Create an OSDK
Create a REST API source for your own Foundry with your OSDK credentials
Use that source in your external transform
But that’s fairly convoluted and adds a fair bit of overhead to each call, so I would recommend just rewriting the calls for the transform (or going with the first option).
If you really want to run this every 5 minutes compute modules might be cheaper (if you run it with very low cpu and memory). You could append the rows to a streaming dataset. I think it’s one of the compute module examples in the walkthroughs.
We’ve additionally recently published (beta) streaming apis that should ease these workflows, if you’re interested in going the streaming route: https://www.palantir.com/docs/foundry/api/streams-v2-resources/streams/publish-record-to-stream/