Does anyone have experience using the combination of:
the S3 sync data connection type
the Foundry S3 API (https://www.palantir.com/docs/foundry/data-integration/foundry-s3-api), being used to do a Foundry-to-Foundry file sync there, on a file-based dataset
I’m trying to figure out what is misconfigured in my setup. I’m getting a CUSTOM_CLIENT exception when trying to finish setup of the connector and preview the S3 bucket (which is the Foundry dataset on another stack). However, I know that the credentials are set up correctly–when I try importing that data connection into a code repo and test it with Boto3, getting temporary credentials based on the configured secrets, it all works correctly. I think the endpoint and url formatting may be off, but after many permutations and switches and tweaks, I’m still getting the same error.
I can’t use the magritte-foundry sync type in this case as an alternative. I’m trying to do this with a file-based dataset, sync it incrementally, filter by prefix, and limit the number of files per transaction. Those kinds of options aren’t in the magritte-foundry sync, since it seems to be built more for tabular datasets.
Here is a redacted version. The CUSTOM_CLIENT exception I’m getting is “Host not found. Please check that you have added the correct egress policies to the source.” But I do have the DNS added for this host as an egress policy, and I also know that importing this source and accessing that same endpoint and bucket via boto3 connects fine.
What do you mean by that? Oh by the way, I should note, I’m trying to connect to another stack–that foundry host url is for the other one. But it is on the egress/ingress lists–we’ve got other data connections and syncs between the stacks, in this direction. Additionally, as I mentioned, in Boto3, with those same paths defined for connection and the same set of credentials, the bucket contents are accessible and match to expectations.
