I have a Workshop application, where my user will filter an Object to a set of 100k to potentially up to 500k Objects.
I want all of those objects to be processed in a pipeline (e.g. trigger an API call for each of them, etc.). What is the best/most efficient way to “record” those 500k objects from Workshop ?
I could think of a few ideas:
Create link between a “set” object type and the actual object, but I can’t perform more than 10k edits in one run
Store the actual string values/etc. so that I can recompute the set in a pipeline via the writeback dataset (but then my users don’t have the full flexibility to filter their object set with whichever property is currently present or will come)
Foundry Rules (but far too heavy for this workflow, given I want a one-off and not a continous evaluaton. I want the “current set” selected by the user, and not the “whichever objects will match those criterias”)
This is a pattern for cohort building I used pretty frequently when running Use Case Boosts - I’m not sure it’s the current state of the art, but it does meet this need quite directly I think.
Everything I wrote before realizing that 'progress' broke my pattern
The missing idea from your list is to save the object set as a resource and then capture the resulting RID as a property on a “cohort” object type. The path to do this without a Webhook and internal APIs is narrow; basically only the Quiver
Form board gives you what you need - a way to take an object set variable and get back an object set RID.
So the pattern I’ve used in the past looks like:
Workshop app where user creates their arbitrary object set based on filters, search-arounds, whatever. An embedded Quiver dashboard that is only a “Submit” button for an Form that creates a new “Saved Object Set” object, which captures some metadata (created at, created by at minimum) and the object set RID that corresponds with the object set variable you pass into the Quiver app.
You now have a nice world where you can kick off an incremental pipeline based on the new object that was created, use an external transform to hit the ontology API, and page through the object set endpoint to “materialize” that particular object set.
As a bonus, you also have a handy “building block” so that users can always navigate back to an object set they defined (the RID property formatting will re-open the saved Object Set in OE).
The main drawback here is that you’re persisting the definition of the object set, at least with the RID, so if you want to keep track of the “snapshot” of exactly which individual objects matched that definition at the moment of materialization, there is a bit more pipeline work to do (and potentially some data scale issues to consider).
Now that Forms has been sunset, there is no longer a Forms board in Quiver, so afaik there is now no way to generate a permanent object set RID for an arbitrary user-defined object set and save that RID as a property on a new object as part of a application workflow (you can do it from OE with the “Save” button), without resorting to Webhooks on internal APIs.