Palantir Foundry Data Connection not Exporting All Parquet File parts

[DCToddM]

I have multiple data connection jobs set up within foundry that exported to an AWS S3 bucket using the Magritte Task. They worked fine for a few years, then at the beginning of June they stopped working for datasets with more than 3 parquet file parts. As an example, if a dataset was only made up of 1 parquet file no problem on the export. Once the parquet had more than 3 parquet file parts it would only export 3 of the parts and then throw this error:

message: com.palantir.logsafe.exceptions.SafeRuntimeException: Attempting to export to the same path a second time:

Anyone have a clue of what this means and how to fix it?

Update 1: We discovered once we took the timestamp off the end of the filename, then all the file parts uploaded successfully. I did send this into the support staff for our instance of Foundry and they are going to look into it.

[malbin]

@DCToddM

I have multiple data connection jobs set up within foundry that exported to an AWS S3 bucket using the Magritte Task. They worked fine for a few years, then at the beginning of June they stopped working for datasets with more than 3 parquet file parts. As an example, if a dataset was only made up of 1 parquet file no problem on the export. Once the parquet had more than 3 parquet file parts it would only export 3 of the parts and then throw this error:

Hey @DCToddM - this sounds like a bug that may require more detailed information about your current setup. Could you file a support issue?

[DCToddM]

I should be able to add in a support ticket. Thanks for the advice.

[aczarnecki]

I would add that we no longer recommend using export tasks. Where possible, you should use new exports, and proactively migrating might be an immediate way to resolve the problem you’re seeing with your export tasks.

New exports (specifically file exports) are available on the S3 source, with more information available in the S3 source documentation.