"ValueError: Only single connection sources are supported." when using get_https_connection in external python transform

[kbitz]

I am attempting to generate presigned URLs for files in an S3 bucket so that I can include them in an export from foundry to airtable (so airtable can then attach in an attachment field). I was unsuccessful using the URLs from the synced media set, so taking this approach instead. However, I can’t seem to even connect to the S3 source within the python code in the first place. I keep getting this error on the code below - I can’t find any documentation as to what constitutes a “single connection” source.

Any help would be appreciated!

Error:

Traceback (most recent call last):
  File "/myproject/datasets/get_s3_urls.py", line 12, in get_s3_urls
    url = s3_scans.get_https_connection().url
  File "/scratch/standalone/c61199ea-623e-4c07-a73a-b4cd3cf8c842/code-assist/contents/transforms-python/build/conda/run-env/lib/python3.11/site-packages/transforms/external/systems/_source.py", line 130, in get_https_connection
    raise ValueError("Only single connection sources are supported.")
transforms.external.systems._redact_credentials_in_output.ValueError: Only single connection sources are supported.

Source code:

from transforms.api import transform, Input, Output
from transforms.external.systems import external_systems, Source

@external_systems(
    s3=Source("<rid>")
)
@transform(
    export=Input("<rid>"),
    output=Output("<rid>")
)
def get_s3_urls(ctx, output, export, s3):
    url = s3.get_https_connection().url
    print(url)

    output.write_dataframe()

[rishir]

Hello @kbitz

This error is usually thrown when there are more than one domains added to your source. Is that the cause with your source?

[kbitz]

When you say more than one domain, what do you mean? I have a single value in the URL field, single value in the S3 endpoint field, and a single egress policy.

In between when I posted this and it was approved, I was able to solve this by hard coding the values needed to create a boto3 client and just using the source for the secret key, however seems like I should have been able to use the source to make the client.

[sandpiper]

I believe that at the present time, the S3 source type does not actually have any associated HTTPS connections for use in external transforms - the “Only single connection sources are supported” exception message is confusing since it is shown when there is more than one HTTP connection or when there are no HTTP connections (I’ll see if we can improve the error message in the “no connection” case).

In general, a good mental model for the current state of the source-based external transform framework (for sources using the direct connection runtime) is that its primary value-add is the ability to bundle the egress policies along with the credentials (and possibly a client certificate) into a single, importable “thing” that can be managed centrally and used in multiple code repositories as desired. The get_https_connection method is just a nice additional convenience for the subset of sources where it works.

[kbitz]

@sandpiper makes sense, thank you for the response!