Implementing a method to export over 200k records from Workshop

Ask the Community
1

I need help with implementing a method to export over 200k records from Workshop.

The user wants to export filtered results from Workshop. Here are their requirements:
・They need to download over 200,000 filtered results from Workshop.
・Ease of use for the user is crucial; complex operations are not preferred.
・The data doesn’t need to be downloaded immediately; notifications upon completion of data build would be acceptable.
・Since similar requests are expected, a generic solution is desired.

I’ve looked at the following method as a reference, but I feel it’s not user-friendly due to the number of user actions required:
https://community.palantir.com/t/how-to-export-more-than-100k-rows-of-a-table-of-object-from-workshop/2708/5

I’m considering a few methods:

  • Using Functions to export filtered ObjectSet from Workshop as CSV.
    → My concern here is that the operation might not complete within the Function’s execution time limit.

  • Using Functions to convert filtered ObjectSet from Workshop into a dataset, then use Transforms to build that data.

    • Method of registering PKs in the dataset
      → Although filtering is applied, retrieving all records takes too long.
    • Method of registering ObjectSet type
      → I’m envisioning data that looks like the following, but I can’t retrieve information from the API
    "objectSet": {
      "type": "INTERSECTED",
      "objectSets": [
        {
          "type": "BASE",
          "objectTypeId": "【base-object-type-rid】"
        },
        {
          "type": "REFERENCED",
          "objectSetRid": "ri.object-set.main.temporary-object-set.【temp-object-set】"
        }
      ]
    }
  • Using Functions to get filter conditions, then apply those filters to the original data in Transforms to build it.
    → The problem here is there’s no way to get the filter conditions set by the user.
  • Implementing pagination to download data within constraints
    → Functions do not provide a way to implement pagination for specific quantities like 100,000 records.

I haven’t come up with a good solution. Do you have any suggestions?