How to export more than 100k rows of a Table of Object from Workshop?

[VincentF]

I want to export more than 100k rows from Workshop, from an Object Table displaying a large set of objects.

How can I export more than those 100k rows ?

[VincentF]

Indeed, CSV and Excel exports are often limited to 100k rows across numerous apps in Foundry.
As a warning: allowing users to export to Excel to support their workflows often shows a misunderstanding of the actual steps users needs to conduct to achieve their work and is very often an indicator that additional discussion needs to happen to really understand “How to best support users”. This can raise potential issues: security controls available in Foundry (markings, data lineage, audibility …) are obviously no longer operating once the data is exported out of the platform.
Please know that Excel can’t support more than 1M rows.

Also, the exact volume of the file will depend on the number of rows, but as well the number of columns and their content. Hence it might be practically very difficult to download a file of even less than 100k rows if the number of columns is high (multiple hundreds) or/and their content is heavy (very long strings, vectors, etc.) as this would generate a multiple-GB final file. Similarly, the time to generate the file increases with the volume of data to export (multiple minutes potentially)

I strongly encourage anyone facing this need to explore in-Foundry alternative, namely:

• Quiver for ad-hoc analysis as a dedicated app (or Contour if operating on Datasets)
• Free-form Analysis widget in Workshop to continue an open ended investigation/filtering/processing of data

Still in some cases (e.g. a third party - like a supplier or customer - not onboarded on the platform that needs to operate on a cut of the data) it might be required to export a large volume of rows.
Multiple options are available:

• Create a transform that generates the CSV/Excel/Any relevant filetype. Writing a transform in a code repository (following those docs) or in Pipeline Builder (repartition to 1 file, file-type of the output to “csv”) will give you a dataset populated with a file, pre-computed, that one can download. The strong advantage of this method is that the size of the file and the time to generate the file can be very large (you could generate multiple GB file, even if the actual file generation takes dozens of minutes). The disadvantage is that the file being precomputed, the content is static and won’t be updated depending on the user selection/inputs.
• Export from Workshop/Quiver/etc. where the export is limited to 100k rows.
• Use Object Explorer where the limit is 200k rows.

On this last option, it is possible to move from Workshop to Object Explorer easily, by the following steps:
In workshop, create a button to “open in Object Explorer”.

In Object Explorer, the object to download the object set will be available top right:

The button can be grayed out if the count of object exceed the 200k limit (by toggling on the “Conditional display” and having a variable transforms that compare the count of current objects with 200k - as an hardcoded value). You can have a tooltip to explain it by writing a description of the button. You likely want to have a metric card (tag format) to show the current objects count.

As a side note, you should consider to use Checkpoints as a way to control and audit who is exporting/what/where, and requiring users to justify the export they perform, if this is relevant for your workflow.
https://www.palantir.com/docs/foundry/checkpoints/overview