Configure Streaming Resources for Data Ingestion

jdemoully
Thu, 31 Oct 2024 03:35:33 +0000
Ask the Community

What is the best way to configure resources for a streaming ingestion? We currently have a Google pub/sub topic we are ingesting with very low data volume. Base on the calculations from the docs, it appears we are using ~8 streaming cores which feels a bit unnecessary and is increasing costs significantly. Note, this is for the raw ingest, not any downstream pipeline builder datasets.

How should I go about reducing the compute utilization for this ingest?

3 posts - 2 participants