Welcome back to our “why we built it” series! Today, we’ll tell you the inside story of Compute Modules, one of our newest developer capabilities.
In our work with various government entities, we often have to deal with satellite imagery data. A satellite flies over a region and captures an image which we then need to divide into tiles and run object detection on. At first, to do this processing, we ran some crazy pipelines with expensive models that needed lots of GPUs, but with spiky loads. The volume of satellite images was low but processing each image generated a very high workload. We needed automatic scaling to make these pipelines performant. At the time, in 2022, we built a custom service that was tailored to exactly this kind of pipeline. However, we wanted to make it possible for our users to own and operate that pipeline in the platform, which couldn’t be done with the custom service.
This custom service was backed by the contour-backend-multiplexer service, some of our oldest infrastructure that does auto-scaling for Contour. It already worked for Spark modules as well, and now we basically had to expand it to other generic containers and expose it directly to users.
The first version of Compute Modules, before it was called that, was just a job spec editor where you could write your deployment spec as a json blob.
The first use case for it was running Python in platform, before we made that easy with Python functions (which by the way, is built on top of Compute Modules!). It was pretty hidden though and used in very few settings.
Although we didn’t really know where we were going with this, we had the vague idea that the flexibility and ease of integrating 3rd party code this would bring to the platform could be useful for other use cases, so we had an intern (who is now a full-time dev at Palantir) build a UI for it as her internship project. If you have an existing model you’ve trained, or a crazy bit of logic written in an old language, you shouldn’t have to spend weeks translating complex code from one language to another!
At the time, this was a very novel idea. Code authoring has always been on rails in Foundry, within certain parameters: specific languages, compute profiles, predefined scale and built-in security. This was a whole new paradigm: the user controls everything. We had to figure out how to make that make sense in Foundry:
- Users can give us any black box of code in a container, and tell us how to talk to it. How do we know if your API is correct when you hit it from within Foundry? What is the user experience of that?
- This gives developers a lot more freedom, but we still need guardrails to uphold our security and data protection guarantees, which are core principles of Foundry.
- How do you make it easy for users to choose what scale they need?
Even then, we had to hunt for the first use cases to test our product thesis. One of those first use cases was a customer that wanted to bring their own proprietary optimization models in a container and run it in Foundry to support scheduling use cases. They packaged their models via Compute Modules and shipped them via Marketplace to a dozens of customers, while benefitting from the custom application building, data governance and compliance certifications that Foundry brings.
A second use case was our partnership with Cohere: their multi-lingual models outperformed others on key dimensions that we needed. We ran their embeddings model within Compute Modules to make it available to our customers.
Since then, we’ve seen project after project use Compute Modules, whether it’s 3rd parties building out models, customers bringing in complex existing business logic or metaprogramming Foundry. This has increased our conviction in the value of “bring your own container” while leveraging the benefits of Foundry interoperability. We spent a lot of effort making the experience palatable for whatever random language you want to show up with, from Java to Julia to Rust.
Today, 60% of compute modules are written outside of Foundry and we want to continue improving this workflow to provide a great user experience, consistent with writing them in Foundry. If you have feedback on this, we’d love to hear it. Since it’s hard for us to tell what’s being done with them, we’d also love to hear examples of how they’re being used and what feature requests you have!
To finish off, we’ll answer a few frequently asked questions…
What has surprised you the most? We didn’t actually expect it to be so popular! There were inklings of developers wanting more pro-code tools for interacting with Foundry, but we had never proven that out. The veritable explosion of interest contributed a lot towards our current major focus on developer experience.
What’s the coolest thing Compute Modules has been used for? Running “space math” in obscure languages to calculate live satellite trajectories is pretty cool! Or another one: extracting measurements and cross-sections from CAD 3D models and using them to generate new images and calculations for various parts of a machine.
Loading the model
Analysis using the Compute Module
A Workshop application with the extracted data
What features are you working on next? Soon we’ll be able to take OpenAPI specs and turn them into Foundry Functions so you can use servers directly without writing adapter code. We’re also working on enabling user tokens to be passed directly into a Compute Module so you scope actions and data down to a specific person’s permissions.
Compute Modules are currently available in Beta everywhere in cloud environments. You can turn it on for a limited group of people in Control Panel, under application access. Developer Tier users should contact Support to request to enable Compute Modules.