Build Failure Conda 403 Forbidden URL

[green]

Starting randomly this morning we’ve had builds failing in one PySpark repository, for a specific Service Account (SA). Every time it starts a (scheduled) build we get a series of errors like below for what looks like all Conda artifacts:

CondaHTTPError: HTTP 403 FORBIDDEN for url <https://waypoint-envoy.rubix-system<snip>/production/foundry-artifacts-api/artifacts/api/repositories/ri.artifacts.repository.cloned.83b83a70-f753-4a85-a9bc-9dab098046a6/contents/release/conda/noarch/python-spark-module-lib-2.340.0-py_0.tar.bz2>
Elapsed: 00:00.055407

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

Additional details:

• The same SA can build datasets in other repos without a problem
• I can build the affected datasets from my account without getting an error
• Same failure if building manually or via a schedule
• Repository is updated to the latest versions

It seems a strange combination of only this account, and this repository, and it was working fine until part way through this morning. We have made no code changes to this repository in the last week. It’s very strange.

[green]

So we figured out the problem.

The repository was importing a code library (a different Python code repository), which was also listed in the meta.yaml file, and the access to that code library was accidentally removed.

Once we removed that import, both under Settings > Libraries and in meta.yaml, the builds could start again.

An interesting confounding problem was that this, now restricted, library was used in two repositories with the same user, but only failing in one. The difference appears to be that the repo that was using a hawk.lock file was unaffected, but the other one that wasn’t using this, but only conda-versions.*.lock files was affected.

I don’t really know enough about the package resolution processes of either to know why this happened, but perhaps the ‘hawk’ based one was running off a cache that meant it hadn’t noticed the loss in permissions??