How does LLM caching work in Pipeline Builder?

Ask the Community
1

hey, three questions on LLM caching inside pipeline builder

  1. does skip recomputing rows for useLLM nodes only work when the temperature is set to 0
  2. how long does the cache live and is there a size limit? x rows times y useLLM blocks
  3. how does this work together with the caching on the platform level for general LLM usage on AIP?
2
  1. skip recomputing rows will work on any temperature
  2. There is no set time or size limit. If you get to any state where you are experiencing degraded performance or issues with time and size let us know!
  3. LMS has their own caching mechanisms, but it’s not as granular as the skip recomputing rows in Pipeline Builder since we keep track of a mapping based on the keys you provide. The LMS cache also isn’t guaranteed to persist