Recently, I have been encountering frequent Out Of Memory (OOM) errors when using LLM nodes in Foundry Pipeline Builder, even for relatively small parsing tasks. While increasing Spark resources (memory, executors) is one solution, I am wondering if there are more efficient ways to design pipelines or optimize LLM node usage to avoid these issues.
-
Are there best practices for minimizing memory usage with LLM nodes?
-
Should I always increase resources, or are there pipeline design patterns that help?
-
Is there a recommended way to preprocess or filter data before passing it to LLM nodes?
-
Any tips for handling large datasets or complex prompts efficiently?
I would appreciate any advice, experiences, or documentation references from the community!
Thank you.