Automated notifications when new patterns are detected

Ask the Community
1

Hi everyone,

I’m working on a data platform tool and have a question about Pipeline Builder capabilities with LLM integration.

My setup:

  • Pipeline Builder with an LLM node that extracts “themes” from text data
  • The LLM outputs structured theme data for each processed record
  • I want to detect when a new theme appears that wasn’t seen in previous pipeline runs

My goal: Send automated notifications (webhook or notification) when the LLM detects a previously unseen theme pattern.

Questions:

  1. Can Pipeline Builder natively detect “new values” in LLM outputs compared to previous runs?
  2. Is there a way to trigger notifications or webhooks directly from within Pipeline Builder when certain conditions are met?
  3. Or do I need to move this logic outside Pipeline Builder (e.g., Functions + Automate, or custom downstream processing)?

Example scenario:

  • Day 1: LLM extracts themes [“Quality”, “Speed”]
  • Day 2: LLM extracts themes [“Quality”, “Speed”, “Safety”] ← “Safety” is new, trigger notification

Thanks!

2

Hey @rsrai , welcome to the community!

Can Pipeline Builder natively detect “new values” in LLM outputs compared to previous runs?

You can achieve this with a dataset that has “append only new values”. Only values that are not already in the output dataset will be put into new transactions (under the hood we get this with an antijoin against the previous transaction on the dataset).

Is there a way to trigger notifications or webhooks directly from within Pipeline Builder when certain conditions are met?

We allow importing java UDFs created in code repos into pipeline builder as a transform under the “reusables tab”; these can be used to run arbitrary code. Though, the credential setup takes some doing.

Or do I need to move this logic outside Pipeline Builder (e.g., Functions + Automate, or custom downstream processing)?

In my opinion the best “foundry-esque” (in its current form anyway) way to do this has pipeline builder output a notification object that has an automate on it which calls a function that does whatever you want.

Anyone else can feel free to add on or add additional notes / context if they feel like it.

3

Thanks @david. I was also experimenting with a different method by creating a function backed action and has automate to find these “New themes” on historical data post LLM processing and then adds the required properties to a notification object type as a new value. The only issue is that I need to create a side effect / webhook to send the notification / email. Maybe this is also the wrong approach so happy to hear any other suggestions!