Add whole folders or projects as References to another project?

[green]

Is there a way of adding multiple Project References at once, or listing all those missing in a repo and then adding them at once?

E.G. you can select at a folder level and it will add all the RIDs of datasets contained in there, or perhaps you can mark a whole project as ‘referenced’ somehow please?

Adding each dataset is very tiresome, and there doesn’t seem to be any way of doing it without either searching each dataset via the ‘references’ button on a project, or going through each transform and clicking the ‘add reference’ high light in the Code Repo (which you can’t search for). Is there any way of listing all unapproved references in a repo?

This is follow up to What are the changes from project scoped exemptions post, where we have now removed them add set the repo to ‘allow security markings’ - but now the checks fail with:

* What went wrong:
Execution failed for task ':checkShrinkwrap'.
> ChecksException: (TransformsGradlePlugin:ShrinkwrapResolutionError): {task="checkShrinkwrap", 
project="<redacted>", errors="\n - Alias 'ri.foundry.main.dataset.<redacted>' has been deleted.\n
 Known files that use this alias: [transforms-python/src/<redacted>, transforms-python/src/<redacted>]"}

These files haven’t been deleted, they just aren’t referenced, so I’m currently having to manually add all these references, which is annoying.

What’s also annoying is that the checks don’t seem to list all the missing files at once. Just a few, so I add them, re-run the checks, fails again with another list of ‘missing’ files etc.

@redboyben I don’t know if the PSJ exemption removal triggered this, or the setting of the markings option in the repo, but it’d be good to highlight as a difference when PSJ exemption is removed.

[nicornk]

Not sure if this helps but if you want to script this, here is code to import a list of rids to a project. From my testing it’s idempotent, which means you can run it again for rid’s that are already imported without breaking things. This would mean you could grab the transforms-shrinkwrap.yml file from your repository, filter for all dataset_rid’s that are inputs and than pass it into the script. Replace the url and token before running:

from typing import List
import requests

def import_rid(project_rid: str, rids: List[str]) -> None:
    """
    Imports the list of rids to the project

    Args:
        project_rid: Project RID
        rids: A list of Foundry Rids to import
    """
    body = {
        "requests": [
            {"resourceRid": rid, "importName": rid}
            for rid in rids
        ]
    }
    _ = requests.request(
        "POST",
        f"https://stack-url/compass/api/projects/imports"
        f"/{project_rid}/import",
        headers={"Authorization": "Bearer token"},
        json=body,
    )

[green]

Ah cool, that’s a very good idea, and I hadn’t thought of doing it via the compass API.

It would be useful to do in the future when we migrate other repos to being Project Scoped again,it saves dealing with the super slow Compass interface for adding references, where you can only search one at a time, you could drop any collection of RIDs into it and have the same effect.

If it’s ok, I won’t accept this as a ‘solution’ (yet, even though it kind of is), in the hope that someone sees the post and thinks it’s a good idea to add that kind of feature natively into Compass/Foundry.

Thanks