We are moving our LLM pipeline to Pipeline Builder from Code Repository. In CR, there is a recommendation to use the NO_EXECUTORS profile for LLM transforms. Is there any such guidelines for PB when running with LLM Blocks? Or can we just run with an Extra Large profile and PB takes care of concurrency and rate limits across executors etc.?
How large is your input dataset? I would be careful about automatically choosing the XLarge profile because you might end up hitting more rate limits and retries (maybe better to start with medium + large depending on size of your data)
Ack.
How would a XLarge profile cause rate limits but not a medium profile? More and bigger executors and cores == higher chance of hitting rate limits?
Yeah with smaller profiles (less executors) there’s not as many concurrent requests.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.