Develop Python library against Foundry dependencies outside of the platform

[pwest-northslope]

Hello Artifact Repositories team,

I’m trying to develop a Python library to be used with python-transforms outside of the platform and running into issues depending on the transforms library.
Using conda-auth (https://conda-incubator.github.io/conda-auth/) in --token mode I can authenticate to the Artifact Repository underpinning a Stemma repository (https://my-stack.palantirfoundry.com/artifacts/api/repositories/ri.stemma.main.repository.{my-repository-locator}/contents/release/conda)
I can then successfully run conda install --override-channels -c https://my-stack.palantirfoundry.com/artifacts/api/repositories/ri.stemma.main.repository.{my-repository-locator}/contents/release/conda transforms.

Unfortunately, adding this artifacts repository to the channels section of my ~/.condarc results in an unexplained HTTP 401 Unauthorized from Artifacts (detailed below).

My conda_recipe/meta.yaml looks like:

# usual `package`, `source`, `test`, and `about` sections

build:
  noarch: python
  script: {{ PYTHON }} -m pip install . --no-deps

requirements:
  build:
    - python =3.9
    - setuptools
  run:
  	- python =3.9
  	- transforms

extra:
  channels:
    - https://my-stack.palantirfoundry.com/artifacts/api/repositories/ri.stemma.main.repository.{my-repository-locator}/contents/release/conda

Running conda build conda_recipe fails with the following error:

Multi-download failed. Reason: Transfer finalized, status: 401 [https://my-stack.palantirfoundry.com/artifacts/api/repositories/ri.stemma.main.repository.{my-repository-locator}/contents/release/conda/noarch/repodata.json] 0 bytes

According to the environment troubleshooting docs (https://www.palantir.com/docs/foundry/transforms-python/environment-troubleshooting#multi-download-failures):

This error means either that you do not have access to the artifacts channel from which packages are downloaded, or that the artifact is not available on the enrollment.

Given that conda install succeeded and that there are no dependencies other than setuptools and transforms, both of which were installed as transitive dependencies of transforms (python is managed by conda), why is this error thrown on conda build?
The same error comes up if I try to run conda-lock lock -f conda_recipe/meta.yaml --lockfile conda_recipe/.conda-lock.yml -p osx-arm64 -p linux-64 -p linux-aarch64 (still the case with only -p osx-arm64 or -p linux-64 instead of all three).

Other things I’ve tried:

• including conda-forge and defaults in the list of channels
• replacing the Stemma repository RID with ri.transforms.python.artifacts.repository or ri.transforms.python.artifacts.runtime
• conda install --override-channels -c {artifactRepositoryURL} transforms → conda env export > environment.yml → conda env update -f environment.yml; this throws CondaAuthError Unable to find authorization token for requests with channel {artifactRepositoryURL}
• ensured that the relevant backing artifacts repositories are set on my Stemma repository (the default ones that a newly-created python transforms repository is initialized with)

Other context:

• The library does need to be developed outside of Foundry
• The library will be uploaded to an Artifacts Repository inside Foundry to be used
• Doing a similar thing with NPM packages works just fine with the Stemma repository as the only NPM registry

Summary:

• developing a Python library outside of the platform with Conda as the package manager & build system
• Using a Stemma repository as a Conda channel
• conda install transforms works; conda build throws an unexplained HTTP 401

[nicornk]

One thing you could do is observe how the Python packages get fetched when you run the local dev gradle task. Gradle has a debug flag and there is quite some information in the logs on the used artifact repositories.

I do think you can also use any code repository rid as artifacts rid and it will proxy through the configured backing artifact repositories.

[jayad]

@pwest-northslope I suspect this is an issue in how conda-build integrates with boa, and given the error message (Multi-download failed. Reason: Transfer finalized, status: 401), I suppose you’ve also been using conda mambabuild.

I found an open issue here that seems to also validate my hypothesis.

I’d recommend switching to use conda build in the mean time so as to unblock you since that one works fine when you’re using conda-auth.

If you still want to use boa as your build backend, you could adjust your ~/.condarc file to include the token directly in the channel URI. Example below:

channels:
- https://:$TOKEN@$STACK.palantirfoundry.com/artifacts/api/repositories/$STEMMA_REPO_RID/contents/release/conda

where you’d need to fill out $TOKEN, $STACK & $STEMMA_REPO_RID. For me, doing so and then re-running conda mambabuild worked successfully.

[jayad]

I looked into putting up a fix for this but i’m afraid it won’t work out for a few reasons. Here’s a walk through if you’re interested

conda-auth is designed to hook around the Conda plugins ecosystem. It both produces a plugin, and adds pre subcommand hooks for Conda to trigger before running commands.

The above works great for any subcommand that also hooks into the Conda plugin system through Python code.

boa (which apparently is now deprecated) makes the mambabuild command available by creating an executable entrypoint, instead of hooking into the Python Conda plugin hook. This is the older way of executing commands, and Conda effectively forks off by executing the binary it’s been asked to run, which makes it impossible to trigger the hooks since the context is lost the moment we jump to the other binary. (ref).

Given boa is deprecated in favour of rattler-build, I would suggest you either move to conda-build or rattler-build.

[pwest-northslope]

Thanks John!
I had tried boa briefly before finding it was deprecated, but never removed the mamba installation that went along with it. Removing that made conda build work nicely with conda auth login https://${STACK}/artifacts/api/repositories/${STEMMA_REPO_RID}/contents/release/conda --token.

Using https://:${TOKEN}@${FOUNDRY_URL}/artifacts/api/repositories/${STEMMA_REPO_RID}/contents/release/conda as the channel URL in either ~/.condarc or extra.channels[] in meta.yaml worked for conda build, but did not work with conda-lock (which I’m also using to get a dev environment from meta.yaml). Substituting the variables with their values in the URL still produces a 401 from conda-lock regardless of where the channel is specified (~/.condarc, meta.yaml extra.channels[], or conda-lock lock --channel) still results in conda-lock getting a 401; it does not seem to properly handle authentication. The conda-auth plugin works nicely with conda-lock now that the rogue mamba installation is gone. Specifying either --no-mamba or --conda /path/to/my/conda/base/envalso did the trick forconda-auth` while that mamba installation was still present.

Thanks!