We have a incremental dataset in pipeline builder, but we rely on window functions to calculate some fields as part of our analytics and final dataset output.
Is it on the roadmap to eventually enable window functions in pipeline builder for incremental datasets? Or does anyone have any workarounds?
1 Like
Hi,
One thing I would do, but I don’t know the full extent of your issue.
- have a pipeline builder, where I would configure the input has a snapshot and would do only the minimal that is needed with the window function.
- have another one, where the incremental transformations are done.
- Have a code repo doing the joins in a incremental way (where we state in the incremental that the 1. is a snapshot, so we don’t trigger a snapshot frequently), since it’s not possible on pipeline builder.
Depending on what you are doing, it might lose a lot of efficient like this, but if it’s a lot of data and complex maybe it’s worth it.