Speed up builder pipelines that use LLM nodes?

Ask the Community
1

Are there any available levers to speed up pipeline builder pipelines that utilize LLM nodes?

2

Hey @agordon we’re currently working on a way to cache useLLM results so that subsequent LLM calls are only done on new rows which should speed things up in the future. I’ll make a note to reply back to this thread when that feature is available!

1 Like
3

Hey, you can try to use a model that has a higher rate limit and is faster such as GPT3.5

1 Like
4

My model options are unfortunately limited due to legal but thanks!

5

Since the driver handles the LLM calls would increasing driver memory and/or cores have an impact?

6

Requests are not made from the driver. The are made asynchronous from the executor. It is very very likely that you’re just hitting rate limits.

1 Like
7

Ahh good to know, thank you!

8

And there’s no way to adjust rate limits, right?

9

Also I should clarify that I’m using a stack hosted model

10

Yes currently you cannot adjust rate limits in the Pipeline Builder UI