Speed up builder pipelines that use LLM nodes?

Are there any available levers to speed up pipeline builder pipelines that utilize LLM nodes?

Hey @agordon we’re currently working on a way to cache useLLM results so that subsequent LLM calls are only done on new rows which should speed things up in the future. I’ll make a note to reply back to this thread when that feature is available!

1 Like

Hey, you can try to use a model that has a higher rate limit and is faster such as GPT3.5

1 Like

My model options are unfortunately limited due to legal but thanks!

Since the driver handles the LLM calls would increasing driver memory and/or cores have an impact?

Requests are not made from the driver. The are made asynchronous from the executor. It is very very likely that you’re just hitting rate limits.

1 Like

Ahh good to know, thank you!

And there’s no way to adjust rate limits, right?

Also I should clarify that I’m using a stack hosted model

Yes currently you cannot adjust rate limits in the Pipeline Builder UI