How can I PGP decrypt ingested files in Foundry?

Ask the Community
,
1

I’m ingesting files that are PGP encrypted (e.g. files that were stored encrypted on a FTP server).

How can I decrypt those files once ingested in Foundry ?

2

You have different options:

  1. You setup a source + an PGP decryption transformers. The files received in Foundry will be deciphered and ready to consume. Lowest complexity of all solutions
  2. You ingest the files as any other ingest (no decryption) and do a transform downstream to decipher the files
  3. You do an external transforms, where you load the files, and decrypt them after their ingest and before writing them to the final output dataset. The files will no be ciphered in Foundry, but you handled all the complexity (connection, PGP, …)

Explanation for [1]
You can use file-based transformers directly in data-connection:
You will need to configure the PGP keys on the agent, and provide the path to those PGP keys (the specific place on the agent machine where the private (or secret) keys are stored)

You can run commands like:

gpg --list-keys
gpg --list-secret-keys

and use the path to the keyring: /path/to/your/custom/keyring/directory/ (not sure if folder or the file directly)

Code snippet for [2]
You can ingest the files encrypted and you can decrypt your files in a transform, by doing something like below.
This is inspired from transforms of unstructured files, and using PGP libs.

@transform(
    out=Output(""),
    source_df=Input("")
)
def compute(source_df, out):
    key, _ = pgpy.PGPKey.from_blob(pgp_key_string)
    for f_name in source_df.filesystem().ls():
        with source_df.filesystem().open(f_name.path, 'rb') as f, out.filesystem().open('new_file', 'wb') as g:
            enc_file_stream = pgpy.PGPMessage.from_blob(f.read())
            plaintext = key.decrypt(enc_file_stream).message
            g.write(plaintext)

Explanation for [3]

You will need to write an external transform and use a library that handles connection to FTPs.

1 Like
3

Hi @VincentF,

I tried to use second option but i have struck on pgp_key_string, i got .asc file for my case.