Hey, I have a streaming transform and I am piping it into an object type. The incoming rows might end up having duplicates amongst them so to avoid Primary Key issues I would like to deduplicate them and only keep the most recent row for each Primary Key.
I looked into using the “Time bounded drop duplicates” function but it only keeps the oldest record instead of the newest. I am looking into using a Window function instead but cannot find docs on the various window methods supported in streaming.
Could someone please provide the docs or suggest what the best approach for this would be?