I’m setting up a Foundry Stream to ingest from AWS SQS & running into issues actually getting data in-platform.
My test setup is an SNS topic which publishes notifications to an SQS queue, an IAM user created specifically to access the SQS queue, and Foundry authenticated to that SQS queue using the AccessKey/SecretAccessKey credentials option.
When I publish a message to the SNS topic, the stream receives a row, but all fields in that row are empty:
I got the schema by inputting the sample JSON of an SNS notification forwarded to an SQS queue to the infer-schema-from-data box; the JSON comes from AWS documentation here: https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html
The logs for the connector do not indicate what schema it is trying to provide to the stream, and I can’t find any documentation with the information.
Live-polling for messages in the SQS console while publishing to the SNS topic does produce the correct message with all of the columns that streaming expects but has left empty.
In the logs for the stream I can see that Data Connection is finding messages in the SQS queue:
Origin: com.palantir.magritte.managedtask.plugin.sqs.streaming.ingest.consumer.SqsQueueConsumer
Message: Received {} messages in the ReceiveMessage request with request ID: [{}] for SQS Queue: [{}].
unsafeParams:
{
"receivedMessages":1,
"requestId":"REQUEST-ID-AS-UUID",
"queueUrl":"https://sqs.MY-REGION.amazonaws.com/QUEUE-OWNER-USERID/QUEUE-NAME"
}
How do I get the fields in the stream to populate?
And is there documentation that I’m missing? The only mentions of “SQS” that I can find in the Foundry documentation state that SQS is a supported source, but provide no instructions for how specifically to configure an SQS source.
Cheers,
Paul