Introducing Python functions for Pipeline Builder, Workshop, and more [Beta]
Python functions are now supported in Pipeline Builder, Workshop, and other Ontology-based applications. Python is a familiar, easy to learn, and well-documented language with an extensive list of libraries for everything from data science to image processing that you can now leverage in the Palantir platform.
Since user-defined Python functions in the Palantir platform are intentionally reusable across applications, you can easily pre-compute values in Pipeline Builder or calculate them in real time in Workshop module when users add inputs. This same function can also be used in other Ontology-based applications to empower decision-making processes for your organization.
The following benefits can be found across the platform when using user-defined Python functions:
- Leverage external libraries: Python has a huge number of libraries that can make development simpler, faster, and more performant.
- Iterate rapidly: Preview your code in Code Repositories as you develop to monitor your results.
Use Python functions in Pipeline Builder
Python functions in Pipeline Builder offer efficiency and flexibility for your pipelines:
- Release with ease: When you are satisfied with your changes, simply tag and release your code to make it available in Pipeline Builder.
- Improve execution time: Reduce your build time by setting your batch size between 100 and 1000, specifying the number of rows to process in parallel.
Add a custom function to your Pipeline Builder pipeline by selecting Reusables in the upper right of your graph. Then, choose User-defined functions > + Import UDF. Here, you can choose the Python function you want to add to your pipeline. Your function will take in a single row, transform it using your logic, and output a single row for all batch and streaming pipelines.
Importing Python functions into Pipeline Builder.
Adding Python functions to the Pipeline Builder pipeline.
Use Python Functions in Workshop and Ontology-based applications
In Workshop, your function will be “deployed”, allowing it to dynamically adapt to the number of incoming requests. During periods of high usage, your deployed function will dynamically scale to support new user inputs.
Preview your function in Code Repositories.
Deployed functions in a Workshop module.
Learn more about Python functions and write your first user-defined Python function with our documentation.
What’s next on the development roadmap?
- A cheaper, faster backend: We are actively working on improving backend performance for Python functions used in Workshop and other applications to match the high-level performance of TypeScript functions.
- Bridging gaps between TypeScript and Python functions: We continue to work to provide the same level of support to Python functions as we do with TypeScript functions. Currently, our highest priority is to provide support for Ontology edits through Python functions.
- Marketplace compatibility: Python functions will soon be available in Marketplace so you can share easily share your functions with other users.