Sort and Limit Functionality in Pipeline Builder

[gcheng]

Hi all,

Wondering if there’s any sort of “limit by top x rows by a certain column” functionality that exists in Pipeline Builder. In this case I’m specifically looking to sort and limit by a timestamp property.

I suppose a potential workaround is to find the timestamp representing the precise percentile of the rows that I must limit by and filter by that timestamp in order to yield the desired amount of rows. However, I don’t believe this is guaranteed to yield an exact number of rows, and this is a significant workaround.

Thanks!

[sperchanok]

Hi there!

The closest thing we have in Pipeline Builder right now is to use the “Top rows” transform, where you can select columns to sort and partition by, and to select the top N rows for each partition.

[gcheng]

Thanks @sperchanok ! Looks like the solution in this case is to treat the entire input as a single partition. I’ll do this via a temporary column for the time being.