Secrets Marshalling is broken in REST API sources

Ask the Community
, ,
1

It looks like on May 13th our Python transform builds started breaking with the following error:

transforms.external.systems._redact_credentials_in_output.JSONDecodeError: Invalid control character at: line 5 column 46 (char 184)
Traceback (most recent call last):  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/_build.py", line 331, in run    self._transform.compute(**kwargs, **parameters)  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/api/_transform.py", line 318, in compute    self(**kwargs)  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/api/_transform.py", line 226, in __call__    return self._compute_func(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/work-dir/__environment__/__SYMLINKS__/site-packages/transforms/external/systems/_redact_credentials_in_output.py", line 24, in wrapper    raise _redact_exception_chain(e, secrets)  File "/app/work-dir/__environment__/__SYMLINKS__/site-packages/transforms/external/systems/_redact_credentials_in_output.py", line 21, in wrapper    return func(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^  File "/app/work-dir/__user_code_environment__/__SYMLINKS__/site-packages/myproject/datasets/transcripts_from_gdrive.py", line 222, in compute    json.loads(secret.encode().decode("unicode_escape").strip('"')),    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/python/json/__init__.py", line 346, in loads    return _default_decoder.decode(s)           ^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/python/json/decoder.py", line 337, in decode    obj, end = self.raw_decode(s, idx=_w(s, 0).end())               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/python/json/decoder.py", line 353, in raw_decode    obj, end = self.scan_once(s, idx)               ^^^^^^^^^^^^^^^^^^^^^^transforms.external.systems._redact_credentials_in_output.JSONDecodeError: Invalid control character at: line 5 column 46 (char 184)

We have an API secrets encoded as a JSON string and a base64 encoded JSON string that, when attempting to decode, all of a sudden started blowing up with this error without any code changes.

And the same thing happened with TypeScript functions. We started seeing the same error attempting to access JSON secrets:

INFO [2025-05-24T15:24:18.126702Z]    SyntaxError: Unexpected token ':'INFO [2025-05-24T15:24:18.126703Z]        at internalCompileFunction (node:internal/vm:77:18)INFO [2025-05-24T15:24:18.126704Z]        at wrapSafe (node:internal/modules/cjs/loader:1288:20)INFO [2025-05-24T15:24:18.126705Z]        at Module._compile (node:internal/modules/cjs/loader:1340:27)INFO [2025-05-24T15:24:18.126706Z]        at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)INFO [2025-05-24T15:24:18.126707Z]        at Module.load (node:internal/modules/cjs/loader:1207:32)INFO [2025-05-24T15:24:18.126708Z]        at Module._load (node:internal/modules/cjs/loader:1023:12)INFO [2025-05-24T15:24:18.126709Z]        at Module.require (node:internal/modules/cjs/loader:1235:19)INFO [2025-05-24T15:24:18.126710Z]        at require (node:internal/modules/helpers:176:18)INFO [2025-05-24T15:24:18.126711Z]        at validateServiceAccount (/app/dist/src/core/index.js:128:29)INFO [2025-05-24T15:24:18.126713Z]        at getGoogleAuth (/app/dist/src/core/index.js:161:35)INFO [2025-05-24T15:24:18.126713Z]  }INFO [2025-05-24T15:24:18.126714Z]}

To fix the issue in the functions layer I removed the code that used the REST API source to retrieve the base64 encoded JSON secret and just hard coded loading it via the file system by including the file in my container (which I do not want to continue doing but have no choice):

const absolutePath = path.resolve(process.cwd(), 'service_account.json');
const fileContents = fs.readFileSync(absolutePath, 'utf-8');
const credentials = JSON.parse(fileContents);
validateCredentialsStructure(credentials); 
console.log('✅ Service account file loaded successfully');    return credentials;

Unfortunately, I can not apply a similar fix to our Python transforms without checking in the secret file to the repo, which I am not willing to do. The secret in question is provided by Google to authenticate signed requests for service users.

Please see what may have changed to cause this error in both our Python transforms and our TypeScript functions. They both retrieve the secrets from the same REST API source, which is why I am blaming a change in that layer of Foundry.

2

Hello, apologies for the trouble here. We’ve recently introduced a change to no longer escape secrets that are loaded from sources when used in code in order to more accurately reflect the values entered into Data Connection.

It looks like you were previously doing some manual unescaping here:

json.loads(secret.encode().decode("unicode_escape").strip('"'))

You should now be able to load your secrets with:

json.loads(secret)

Can you let me know if this fixes the issue for you?

1 Like
3

Thanks for the reply Jett. I will attempt to implement it today.

4

I refactored this function:

def get_credentials_from_secret(google_connection_source, logger, subject_email):
    # Retrieve the secret JSON string containing Google service account credentials
    secret_json = google_connection_source.get_secret("additionalSecretJsonSecret")

    # Decode the JSON string to handle escape characters and strip quotes
    decoded_string = secret_json.encode().decode("unicode_escape").strip('"')

    # Load the decoded string into a JSON object (dictionary)
    json_data = json.loads(decoded_string)

    # Create credentials using the JSON data and specify the required scopes
    credentials = service_account.Credentials.from_service_account_info(
        json_data,
        scopes=["https://www.googleapis.com/auth/gmail.readonly"],
        subject=subject_email,
    )

    return credentials

To this:

def get_credentials_from_secret(google_connection_source, logger, subject_email):
    # Retrieve the secret JSON string containing Google service account credentials
    secret_json = google_connection_source.get_secret("additionalSecretJsonSecret")
    
    # Load the decoded string into a JSON object (dictionary)
    json_data = json.loads(secret_json)

    # Create credentials using the JSON data and specify the required scopes
    credentials = service_account.Credentials.from_service_account_info(
        json_data,
        scopes=["https://www.googleapis.com/auth/gmail.readonly"],
        subject=subject_email,
    )

    return credentials

And this code:

def compute(ctx, raw_meeting_transcripts, google_connection_source):
    """Main transform function to collect and store transcripts."""
    logger.info("Starting transcript collection")

    # Initialize credentials first
    secret = google_connection_source.get_secret("additionalSecretJsonSecret")
    creds = service_account.Credentials.from_service_account_info(
        json.loads(secret.encode().decode("unicode_escape").strip('"')),
        scopes=["https://www.googleapis.com/auth/drive"],
    )

To this:

def compute(ctx, raw_meeting_transcripts, google_connection_source):
    """Main transform function to collect and store transcripts."""
    logger.info("Starting transcript collection")

    # Initialize credentials first
    secret = google_connection_source.get_secret("additionalSecretJsonSecret")
    creds = service_account.Credentials.from_service_account_info(
        json.loads(secret),
        scopes=["https://www.googleapis.com/auth/drive"],
    )

Running build now.

5

Looks like the fixed worked! Thank you again. Hyrums Law: “With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.”

6

Glad you were able to get things working!

I’d like to apologize again for the disruption and thank you for flagging it. We recognize that breaking changes are unacceptable and can interrupt critical workflows. Following this incident, we are introducing stricter processes around performing upgrades and changes like this in the future. We can and will be more vigilant about making sure our users don’t end up in these situations in the future.

2 Likes