im trying to extract unique entities from a streaming timeseries dataset. so i’d drop the timestamp and want the output to be a dataset deduped on the entity_id. im not seeing a way to either drop duplicates without a timestamp or even output the dataset as incremental or some way to dedupe on that key. any ideas?
Are you able to use the archive dataset for the stream to process and dedupe? Link to docs: Cold Buffer