Streaming Pipeline Performance Dependent on Number of Outputs?

dramesh
Wed, 05 Jun 2024 13:57:26 +0000
Ask the Community

Hi Folks,

I’m trying to understand the influence the number of streaming outputs I have in a single streaming pipeline has on overall streaming performance/latency for each output.

For example, let’s say I have three different streaming outputs in the same pipeline builder which do not share any logic (i.e the individual pipelines for each individual output do not share any builder boards). If one of the outputs gets updated at a much higher frequency than the other two, is it possible that it’s updates “crowd out” the new events for the two slower updating outputs since they all share the same streaming queue (from my incomplete understanding)?

My current build resource profile is small and I would like to keep it to that.

5 posts - 3 participants