Raw Amazon Kinesis multi-table stream to formatted streaming datasets

Ask the Community
1

Are there any best practices/out-of-the-box solutions to take a raw Amazon Kinesis stream with multiple tables of changelog data and turn it into a per-table CDC stream? Ideally it would handle DDL changes as well.

2

There’s no OOTB solutions for this specific setup. Here is how I would do it using the tools Foundry has:

  1. Use the Kinesis connector to sync your stream into Foundr
  2. Create a streaming pipeline in pipeline builder
  3. In the streaming pipeline, separate the stream into separate outputs based on which table it comes from
  4. Use the Key By transform to enable CDC mode on each of the output streams

As an alternative, Foundry does support native CDC capabilities if you are connected directly to the database instead of reading from a kinesis stream.

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.