Are there any available levers to speed up pipeline builder pipelines that utilize LLM nodes?
Hey @agordon we’re currently working on a way to cache useLLM results so that subsequent LLM calls are only done on new rows which should speed things up in the future. I’ll make a note to reply back to this thread when that feature is available!
Hey, you can try to use a model that has a higher rate limit and is faster such as GPT3.5
My model options are unfortunately limited due to legal but thanks!
Since the driver handles the LLM calls would increasing driver memory and/or cores have an impact?
Requests are not made from the driver. The are made asynchronous from the executor. It is very very likely that you’re just hitting rate limits.
Ahh good to know, thank you!
And there’s no way to adjust rate limits, right?
Also I should clarify that I’m using a stack hosted model
Yes currently you cannot adjust rate limits in the Pipeline Builder UI