We are looking to optimize the split of our vCPUs between Default Queue and User Home Folder Resource Queues and are wanting to understand how many to keep in the User Home Folder Queue.
To my understanding Pipeline Builder Preview uses the Extra Large Profile?
Does that mean that the preview’s use 257 vCPUs…? (driver + executor cores * num executors) Or does it generally use a smaller amount and somehow scale up to 128 max…?
Hey Max, not sure I fully understand the question, the preview spark module is shared between all users.
We also configure it to use dynamic allocation so the number of executors will dynamically scale up and down over time.
Hey Xander, I was trying to understand on average how many vCPUs to provision to user home folder resource queue (where to my understanding the two main sources of Compute here are PB Preview and Contour)…
Therefore was looking in to trying to estimate if we have x concurrent builders running pipelines, how many vCPUs is that likely to use?
To do so my thought process was 1. what profile does PB Preview use (I think extra large?) 2. how many vCPUs does that actually equate to (seems this in itself is also dynamic based on previous previews ran in this pipeline?)