Is it possible to ingest data from a Kafka topics as batch datasets instead of streams?

Ask the Community
,
1

I am ingesting data from a Kafka topic into Foundry Streams. But I don’t actually need the near real-time update capabilities offered by Foundry Steams in this particular instance.

Is there a way to ingest data from a Kafka topic on a schedule (e.g. every 5, 15 or 30 mins) into regular datasets using batch compute (w/o every utilizing Flink)?

2

hey @acapras

Consuming Kafka as a stream is the most efficient way and why we only support this ingestion method.

If you don’t need to have all of your downstream pipelines to be real-time, our recommendation is to use the archive dataset (also called cold buffer) as a batch input to your pipeline.

2 Likes