Streaming Pipeline Performance Dependent on Number of Outputs?

Ask the Community
,
1

Hi Folks,

I’m trying to understand the influence the number of streaming outputs I have in a single streaming pipeline has on overall streaming performance/latency for each output.

For example, let’s say I have three different streaming outputs in the same pipeline builder which do not share any logic (i.e the individual pipelines for each individual output do not share any builder boards). If one of the outputs gets updated at a much higher frequency than the other two, is it possible that it’s updates “crowd out” the new events for the two slower updating outputs since they all share the same streaming queue (from my incomplete understanding)?

My current build resource profile is small and I would like to keep it to that.

2

When setting your build resource profile, that applies to the entire pipeline. Thus, if you have multiple different streaming outputs, they will all be sharing the same resources specified by your profile, and if you have a very “active” stream that has a higher throughput, it may consume more resources than the less active streams. Notably, this resource contention is not because the outputs all share the same streaming queue, but rather because the amount of resources configured is fixed across the pipeline. The only way to truly isolate a higher-throughput pipeline from other pipelines is to create it separately as a new resource, this way you can more finely tune the resources necessary for each pipeline.