We have an incremental batch pipeline and want input rows to be processed row-by-row since an update to the second row is dependent on the update on the first row, and so on. What’s the best way to do this?
It sounds like a streaming pipeline might be what you want here. If you need to keep it as an incremental batch pipeline you could also have a timestamp column that let’s you keep track of the ordering of when the rows came in and go from there.
If neither of those work feel free to give more details on what you’re trying to calculate