Hey folks,
I have an external endpoint that produces data at the sub-5 second scale. The only way to interact with the endpoint is through an API call. I currently do this through an external transform that builds every minute. I would like to convert this into a streaming sync. Any ideas on how I could achieve this?
Unfortunately, this is not currently possible.
But can he run the poll every minute and then, in the transform, push the new records to a stream? So the rest of the pipeline is a stream?
Since this question was asked, compute modules were released. This is something you should be able to accomplish with a compute module since it’s long-running.