Is there a way to programmatically get the list of all datasets that are in a compass folder, in code repo, and store this list in a list?

Ask the Community
1

I am trying to write a transform generator that iterates across all datasets in a compass folder. How can I get the list of dataset names that exist in a given folder?

2

Hi @jcosta,

There is currently no nice way to do this automatically in Foundry, however the best way I’ve found to auto generate your transforms_data for a transform generator from a folder is:

  1. Inspect the network calls that load the Compass data when you load the folder page in
  2. Look for the network call that hits /compass/api/folders/<your_folder_compass_rid>/children
  3. Copy and paste the response data into your favourite LLM or AIP assist in Foundry and ask it to format the dataset RIDs into your required transform_data shape

That should do the trick, but let me know if you run into any issues!

3

Hello - You can actually find the documentation for this endpoint at

https://{{your.stack.url}}/workspace/documentation/developer/api/compass/services/CompassService/endpoints/getChildren

4

You will also need to configure your transform as an “external transform”, and set up egress and credentials.

Please don’t use your own API Token to authenticate your requests, as it could be abused to impersonate you. Instead you might want to consider setting up a service user via the Dev Console.

5

You could use foundry-dev-tools. The CompassClient has an implementation to get all children of a folder:

https://emdgroup.github.io/foundry-dev-tools/api/foundry_dev_tools.clients.compass.html#foundry_dev_tools.clients.compass.CompassClient.api_get_children

You can run fdt within foundry as external transforms - [docs link]https://emdgroup.github.io/foundry-dev-tools/configuration.html#configuration-for-transforms - or just from your own local machine.

from foundry_dev_tools import FoundryContext

ctx = FoundryContext()

children = [
    {"rid": entry["rid"], "name": entry["name"]}
    for entry in ctx.compass.get_child_objects_of_folder(
        "ri.compass.main.folder.12da4293-7457-4042-9d43-c64e3e54ea36"
    )
]
print(children)

will return

[{'rid': 'ri.foundry.main.dataset.43dea6c7-58ae-40b6-96ed-ddaaf216d003',
  'name': 'dataset-name1'},
 ...
 {'rid': 'ri.foundry.main.dataset.1263dc7b-665a-48ef-8dcf-9da57ade6fdf',
  'name': 'dataset-name2'}]```