How do I create a code block in pipeline builder?

Ask the Community
1

How do I create a code block in pipeline builder? I can’t seem to find it in the transform options

2

Hi,
Pipeline Builder doesn’t support code blocks directly in the UI.
We support user defined functions (UDFs), but these are authored in code repositories, then published and imported into a pipeline.
See documentation for more info.

3

Expanding on this for the end to end solution:

  1. Create a new code repository
  2. Select “Functions” in the the repo creation UI
  3. Select “Python Functions” in the next UI page
  4. Create a python file in python-functions/python/python_functions
  5. Write your function which takes as inputs individual values, not containers such as pyspark columns. Make sure that you add type annotations to all your function parameters and the return type.
  6. Commit your code
  7. Tag your code and release it
  8. Wait for the release to complete
  9. Switch to your pipeline builder
  10. In the top left click the drop-down that says “reusables”
  11. Click “user-defined functions”
  12. Click “import UDF”
  13. Add the UDF you have created
  14. Create a transform path in pipeline builder
  15. You can now find your function amongst the available transformations.
1 Like