I’ve built out some knowledgebase tools whose backing data is sourced from our CRM. I’m
using these tools to take new inbound customer service queries and generate potential solutions based on the backing data.
These tools are workshops sheets backed by logic functions and a couple of Agents built in Agent Studio.
The backing data contains case histories and knowledge articles in plain text as well as their embeddings, along with some metadata about the cases and knowledge articles.
Included in the backing data are two properties, Case Number and Case URL, I use these to link the generated responses back to our CRM for reference.
Both the agent and the logic functions are performing semantic searches, then returning relevant objects including the case/article text, Case Number and Case URL.
In the prompts for both the logic functions and agents, I ask for the model to link back to the original cases using Markdown and give examples of the formatting using the property names - [Case Number](Case URL)
Queries going to the logic functions reliably return correctly formatted links back to the CRM
The Agents are hit and miss. Sometimes every suggested solution has a correctly formatted case link, some times one or two suggested solutions has a link, sometimes none do.
I’ve tried adjusting my prompts in many different ways but nothing seems to make the Agents behave the same way as the logic function generated response.
Is this just an idiosyncrasy of the Agent Studio? The functionality/inputs are effectively the same between logic functions and the semantic search the Agents are doing.
If you’re looking to make the links inline with the rest of the generated text, I’ve had some luck breaking this apart into separate LLM blocks in my logic: the first generates the text and the second has the URL as an input and the only instruction is to review the text and insert the link as Markdown in the appropriate spot. I’ve found this more reliable than trying to have it both generate the response text and properly encode the markdown url in the same context window.
If you’re happy having the link as a footnote / source at the end of the text, I’d have a non-LLM block (either a Typescript / Python Function or a couple of Logic blocks strung together) to create the correct markdown format deterministically and append it to the generated response.
So far, I’m not having any issues with outputs from logic functions inserting the URL using Markdown format. All my logic functions seem to handle this easily using a single block that uses the properties output from a semantic search.
The issue I’m having is getting an AIP Agent to do the same. Both use the same input properties from the same ontology objects.
No matter how I set up the system prompt for the agent, having it insert inline markdown combining the URL/Case Number is really hit or miss.
The only potential solution I’ve come up with is that I might need to pre-generate the correct URL/Case Number markdown in pipeline when processing the case/article ontology then feed that in as a property the agent can use when generating responses.
Hey @Samwise_AIP ! Dev on AIP Agents here – sorry to hear that you’re experiencing trouble getting your Agent to output the proper citations! Instead of using the markdown format, I would try using the citation format described in the docs here.
If I were you, I would use the AIP Logic function to output the citations in the format <citation><name>Case Number</name><href>Case URL</href></citation> which adheres to the schema detailed in the docs. I would then configure the AIP Agent to use function-backed context (see docs here) which can call the AIP Logic function under the hood and therefore insert the custom citations into the prompt.
You also don’t have to use function-backed context to achieve this – you could also just write a system prompt for your AIP Agent that asks it to output citations in the format <citation><name>Case Number</name><href>Case URL</href></citation>. I would advise against this if you also have Ontology context configured on your Agent because then the Agent will default to the citation format for Ontology objects provided out-of-the-box. Check out the raw system prompt in the View Reasoning panel to see what I mean!
Let me know if this works for you! Otherwise, I’m happy to help support further!
I’ve been reading through the documents and I think I’m clear on the what, but I’m struggling with the how, everything I’ve built so far in Foundry has been no-code, I’m what you guys call an NCE
Is there any chance we could connect to chat about this?
