Schedule Data export as trigger for API call

Ask the Community
, , ,
1

We have a use case for which we need to export an XML file to a S3 bucket (Data export) and then call an external API (transfer_api) from code repository that will push the file on the bucket to a consumer.
Our point is, as it’s not possible to trigger “transfer_api” when the “Data Export” is done as a dataset is expected as a trigger in the schedule, how can we do to launch “transfer_api” only when the export is done?
We have a way in Code Repo to check the content of a S3 bucket to check if the file is present or not and then run the tranfer_api. But if we call “transfer_api” when the export isn’t finished/done, we will have an issue because the file expected won’t be present on the S3 bucket.

So, I’m looking for suggestions/ideas which can be useful in this scenario.

2

Side Note: This used to work with data export tasks because those were regular datasets but those have been deprecated.

What you can do today is use the s3 source in code repo, get the AWS credentials from the source (something like source.get_aws_credentials() not sure this is documented by now), construct a boto3 client with it and upload the object to s3 with an API call from code.

Afterwards, in the same transform, you can call the other API to signal the file is updated.