Hello, We are using a ‘large’ PyTorch model as part of our pipeline and we are having long build time (around 1hr) to just infer 1 or 2 rows. While, when using a live deployment using same architecture (1*T4 GPU), the inference would take a few dozen of seconds. Is there particular points to pay …

Hi, For Spark with sidecar, I would recommend using the ARROW_ENABLED profile (see Spark profile reference ). I expect most of the slowness with Spark comes from having to convert the dataframe from Spark to Pandas format, which is generally slow and memory intensive, but made much more efficient wi…

~seconds vs hours feels like you might be running inference on CPU rather than GPU… I’d check that the GPU is indeed with something like that: import torch import logging from transforms.api import transform, Output, lightweight @lightweight(gpu_type='NVIDIA_T4') @transform(out=Output('/Project/fo…

Continuing this thread as I am still experiencing performance issue on CPU On a 350mb DistilBERT torch model: Live deployment (0.5 cpu - 2GB): 1000 similar rows - 11sec for inference once deployment is ready. (=approx. 20min for 100,000rows) Build Lightweight (2 cpu - 8gb): 1000 rows - OOM Buil…

Hello, So yes, we ended up using this kind of configuration of lightweight transform: A Lightweight transform with lazy polars + quantized ONNX (from the base Pytorch) yielded great results: For 100k rows: from multiple hours with a mid-sized Spark profile (with distributed inference wrapper) dow…

Palantir Developer Community

Huge difference between Live inference & Build with a model

Ask the Community