Logs from UDFs in Pipeline Builder

I have a Java Map UDF running in a streaming pipeline. In my Java code, I have a SafeLogger that I use to output some debugging data. However, when I download the logs from the build screen, I do not see any logs from my UDF in the downloaded file.

Is there anything else I need to configure to get logs from my UDF?

1 Like

It sounds like you have set up everything that is necessary to log output from your UDF in a builder pipeline. Based on my testing, it appears that logs are properly being displayed when run as part of a batch pipeline, but not as part of a streaming pipeline.

This is not expected and is a bug that we are currently investigating. In the meantime, if you would like to confirm that logging is correctly set up within your UDF, you can run it as part of a batch pipeline and see if the logs appear there. If so, then once this is fixed, you can expect them to appear in your streaming pipeline logs as well.

Thanks for surfacing this issue, and we will let you know once it has been resolved!

Hi, any update on this? I also have a Java Map UDF, and I have a branch with this setup

        try {
            throw new Exception("something went wrong!");
        } catch (Exception e) {
            System.out.println(e.getMessage());
            logger.error("what are you doing!" + e.getMessage());
        }

Even when I use the UDF in a batch pipeline, I don’t see any logs in the browser or when I download them.

Hi, could you share more detail about how you’re constructing this logger? I was able to see logs from a UDF running as part of a batch pipeline with the following setup:

  • Adding: implementation 'com.palantir.safe-logging:logger' under the dependencies section of the subproject’s build.gradle file
  • Constructing the logger as: private static final SafeLogger log = SafeLoggerFactory.get(MyMapUdfImpl.class);
  • Logging as: log.warn("Hello world"); from inside my UDF logic

If you try the above and it still doesn’t work, any other details you can share about how I may be able to reproduce this behavior would be much appreciated!

Sure, here is some extra info:

How I created the UDF:

  • create new code repository
  • choose “pipelines”
  • click on “UDF” Tab, then select “Java Map UDF Definition”

Left everything as is in the new UDF, except added your changes. Build was successful.

Created a new batch pipeline and used the UDF.
There are still no messages in the pipeline preview or when I download a txt of them.

I also checked the logs of the pipeline deployment for my message and couldn’t find it there either.

I don’t believe UDF logs are surfaced in preview, you would need to deploy the pipeline to view the logs.