How to provide front end users and interface where they can test changes to an ontology object property

Ask the Community
,
1

I’m seeking to provide front end users the ability to test a llm prompt that is buried in a pipeline.

The llm uses a property on a object. For example indicator , when the indicator changes i want them to be able to test the output

I was thinking of an aip logic function in a workshop app the mirrors the llm prompt in the pipeline.

How have others done this ? The users are somewhat tech savvy

2

Hi @AIFDE, from your writeup I am assuming this is a useLLM node in your pipeline that you would like to run evals on.

If that is the case I’d recommend trying out Pipeline Builder Evaluation Suites! Linking some of our docs on this feature below. Please let me know if this is what you are looking for and if you have more questions.

Docs - https://www.palantir.com/docs/foundry/pipeline-builder/evaluation-suite

Thanks,

Alex

3

@AIFDE

your idea is the usual route. a logic function can be used in workshop like any other function, and as long as it only returns text (no ontology edits) you don’t even need an action

what’s worked well for this:

  • build the logic function with the same model and prompt as the pipeline llm step. inputs: the object, plus a separate indicator parameter
  • in workshop add a text/number input for the indicator and a button that runs the function, output into a string variable shown in a markdown or text widget. users try different indicator values without touching the real object
  • seed it with real cases: in logic’s Run panel you can save inputs as unit tests, and Run history > Add as test case turns past runs into an eval suite. AIP Evals can also generate one for you

two things to watch:

  • you now have the prompt in two places, so they will drift. treat the pipeline one as the source and copy changes over, or the other way round, but pick one
  • from workshop the function has a 5 minute execution limit (the logic debugger doesn’t), so keep the test prompt lean