Hello,
I came across this interesting (but short) documentation regarding the implementation of a proxy or federation layer (I think mostly call model router outside the platform?).
Bring your own model • Build a proxy or federation layer with AIP • Palantir
While there is a tiny example mentioning the deployment of a container, additional documentation would be welcomed ?
For example, we successfully deployed a 4B model asset as a container with vLLM engine but one thing we couldn’t do is to have the weights of the model loaded from a dataset ? Today, the weight are backed in the image. As no persistent storage is available, at every spin up we need to reload everything which takes around 5-10min while it would be better to have the model weights, CUDA graph etc.. stored in a dataset. This would allow:
- Lighter docker image
- Possibility of hot swap/sleep of model weights
- maybe GPU Release ?
- and more…
Regarding the routing, where the router layer would be deployed ? In its own Compute module with read permission to the different models (Self-hosted , Palantir provided, External provided)? This would be the endpoint to use when doing BYOM for AIP features? Can this endpoint be tied to a custom endpoints so it can be reached from outside of Foundry ?
As Palantir partnered with Nvidia, can we expect some kind of NeMo Switchyard examples?
Our Docker files, hope it could be useful to the community - Model name redacted:
Dockerfile for llama.cpp with baked-in model
Optimized for GPU (Nvidia A10) batch inference in Palantir Foundry sidecar deployment
Based on: ghcr.io/ggml-org/llama.cpp:server-cuda
FROM ghcr.io/ggml-org/llama.cpp:server-cuda
USER root
WORKDIR /models
Create non-root user (UID 1000) for Palantir Foundry security requirements
RUN getent group 1000 || groupadd -g 1000 modeluser;id -u 1000 || useradd -u 1000 -g modeluser -m modeluser
Copy model files first, then chown after user/group exists
COPY model_checkpoint/Model-3/Model-3-Q4_K_M.gguf /models/COPY model_checkpoint/Model-3/mmproj-BF16.gguf /models/
Set ownership on model files
RUN chown 1000:1000 /models/*.gguf && ls -lh /models/ && echo "Model files baked successfully"
ENV LD_LIBRARY_PATH=/app:$LD_LIBRARY_PATH
USER 1000
EXPOSE 8080
ENTRYPOINT ["/app/llama-server"]
──────────────────────────────────────────────────────────────────────────────
CMD — optimized for Nvidia A10 GPU batch VLM extraction
──────────────────────────────────────────────────────────────────────────────
CMD ["-m", "/models/Model-Q4_K_M.gguf","--mmproj", "/models/mmproj-BF16.gguf","--host", "0.0.0.0","--port", "8080",
Context size (large for extraction tasks)
"-c", "12288",
Use all available GPU layers (A10: 24GB VRAM, Q4_K_M fits easily)
"-ngl", "99",
Parallel slots for batch inference (tune for workload)
"--parallel", "4",
Continuous batching
"--cont-batching",
Prompt batch sizes (tune for A10, large batch for vision)
"-b", "4096","-ub", "1024",
CPU thread tuning (minimal, most work on GPU)
"-t", "4","-tb", "2",
Flash Attention (auto, recommended for Q4_K_M)
"--flash-attn", "auto",
Cache quantization (Q8 for max throughput)
"--cache-type-k", "q8_0","--cache-type-v", "q8_0"]
vLLM:
FROM nvcr.io/nvidia/vllm:26.05.post1-py3
USER rootWORKDIR /models
Create non-root user required by Foundry
RUN getent group 1000 || groupadd -g 1000 modeluser;id -u 1000 || useradd -u 1000 -g 1000 -m modeluser
--- DEBUG LOGGING ADDITIONS ---
Enable verbose debug logging for vLLM
ENV VLLM_LOGGING_LEVEL=DEBUG
Force CUDA errors to be reported immediately
ENV CUDA_LAUNCH_BLOCKING=1
Log GPU communication details
ENV NCCL_DEBUG=INFO
--- OFFLINE MODE ADDITIONS ---
Force HuggingFace libraries to stay offline so they don't timeout trying to ping the hub
ENV HF_HUB_OFFLINE=1ENV HF_DATASETS_OFFLINE=1
Disable vLLM usage stats just in case it tries to send telemetry
ENV VLLM_NO_USAGE_STATS=1
-------------------------------
Copy model files and set ownership in one step
COPY --chown=1000:1000 model_checkpoint/Model-W4A16 /models/Model
USER 1000EXPOSE 8080
ENTRYPOINT ["vllm", "serve"]
Added "--trust-remote-code" to the arguments
CMD ["/models/Model3","--trust-remote-code","--limit-mm-per-prompt", "{"image": 30, "video": 0}","--chat-template-content-format", "openai","--generation-config", "vllm",
"--max-model-len", "131072",
"--gpu-memory-utilization", "0.95","--speculative-config", "{"method": "mtp", "num_speculative_tokens": 2}","--host", "0.0.0.0","--port", "8080"]