Perform 100k API calls at scale, from dynamic user selection

Ask the Community
, ,
1

I want to let my users select a bunch of objects (100k-10M) from a Workshop application and for each of those object, one API call shall be executed.
The API call is very small and quick, and the response might need to be written back to the same object or another one.

What is the simplest way to achieve this workflow in Foundry ?
Note: The workflow is particular challenging because of the “dynamic” nature of the selection.

The simplest and most robust I can see is to use this community post to select objects and then use writebacks/materializations in addition of External Transform, so that each row representing an object is processed as part of a transform.

This has some downsides (a bit complex for a new dev to set this up, as this touches a few parts of the platform; need to handle the incrementally, need to make the external transform resilient to failures, etc … ) hence maybe there is another approach that would be simpler.

e.g. How would doing this in Automate would work ? Would that be efficient ? In how much time can we expect the API calls to be performed ?

Bonus point: If this is possible to set “when” should the API call be sent (e.g. at 1h granularity).

2

Haven’t done this before, but I think you could leverage automate as follows:

  1. add an edit-only property called execute to the object
  2. have an action that can set this property to true
  3. in workshop, allow the user to select any arbitrary number of objects and run the action on it
  4. Create an automate that evaluates the object set (every x amount of time)
  5. When that automate is triggered - run a function backed action to hit the api and write it back to the object.

Unknowns:

  • If your api has a batch endpoint, you could hopefully call this in batch. I am not sure how to configure an automate to call the action one time per batch.
  • API Call times - I think this would depend on rate limiting, number of batches, and evaluation criteria (i.e. run every time the objects get changes, every x amount of time etc)
1 Like