Prompt - Invalid Object Query Tool Input - AIP Agent Studio

Ask the Community
1

When running some prompts against the LLM the prompt works but sometimes I get this error –

Retrying “Object Query Tool” tool call (failed tool input parse). Correction message: "Tool failed to parse input, check the format: Invalid Object Query Tool input syntax: json { "OBJECT_TYPE": "XXXXXXXXXXXX", "FILTER": [ "clli8 == 'XXXXXXXX'", "property value IS NOT NULL" ] } "

What would cause the inconsistency in the answer?

When I start a new AIP Agent Studio session or refresh page it usually works and then reverts to the error message.

Thanks.

2

Whenever the LLM calls a tool there is some nondeterminism in whether the LLM will format the tool call response correctly. If the response is slightly different than the format we expect, we will fail to parse the input to the call and provide the error response to the LLM. In this case, you should see the agent “retry” the tool with corrected formatting. Does the agent always fail to parse the Object Query Tool?

If so, could you paste an example of what the raw response was from the LLM for that iteration in the chain of thought? You can grab this information from the “view reasoning” button for the message, scrolling to the tool call in the reasoning, and clicking on the “show raw” button.

It should look something like this:

Thought:
- I need to query the NYCBuildings object type to find the tallest building.
- To find the tallest building, I should order the buildings by their heightroof property in descending order.
- I should limit the result to 1 to get the tallest building.

Tool Name: Object Query Tool
Tool Input:
OBJECT_TYPE NYCBuildings
ORDER BY heightroof DESC
LIMIT 1

If you notice that there is some piece where the agent is consistently failing to format correctly, you can provide additional prompting in the instructions to reduce these cases.

3

The agent sometimes performs and returns the correct answer. If I try the run the prompt after a successful run, it usually fails.

Thought:

  • I need to ensure the correct syntax for querying the ‘xxx’ object.
  • I will filter for entries where “xxx” is not null and specify the properties to retrieve.
  • I will correct the syntax and include the properties to retrieve in the query.

Tool Name: Object Query Tool
Tool Input:

OBJECT_TYPE xxx 
FILTER xxx IS NOT NULL

The error is - Retrying “Object Query Tool” tool call (failed tool input parse). Correction message: "Tool failed to parse input, check the format: Invalid Object Query Tool input syntax: plaintext OBJECT_TYPE xxx FILTER xxx IS NOT NULL "

I am wondering what the “plaintext” comes from in this particular case.

Another case for the same prompt but requesting more filters:

Thought:

  • I need to correct the query syntax for the ‘xxx’ object.
  • I will ensure the property names match exactly with those available in the object properties.
  • I will filter for non-null “xxx”.
  • I will filter for the city “MIAMI”.
  • I will filter for the state “FL”.
  • I will retrieve the properties “clli8”, “address”, “city”, “state”, “geohash”, and “region”.

Tool Name: Object Query Tool
Tool Input:

OBJECT_TYPE xxx
FILTER xxx IS NOT NULL
FILTER city == "MIAMI"
FILTER state == "FL"

Error - Retrying “Object Query Tool” tool call (failed tool input parse). Correction message: “Tool failed to parse input, check the format: Invalid Object Query Tool input syntax: OBJECT_TYPE xxx FILTER xxx IS NOT NULL FILTER city == "MIAMI" FILTER state == "FL"”

4

I am also confused to where the plaintext is coming from as I have not seen that before. Is the tool input piece wrapped in backticks ` or quotations "? What model are you using?

5

backticks - ```plaintext OBJECT_TYPE - the model is GPT-4o

6

Can you try adding some additional prompting like “when using the Object Query Tool, DO NOT wrap anything in backticks (```)”?

How frequently does this LLM include these backticks? We may be able to add some additional parsing to remove backticks from the input block when they are present. I’ll flag internally for us.

7

I will try the additional prompting.

Backticks are present on many occasions in the thought process as it tries to format the input and Object query failed is the result.

Retrying “Object Query Tool” tool call (failed tool input parse). Correction message: "Tool failed to parse input, check the format: Invalid Object Query Tool input syntax: plaintext OBJECT_TYPE XXX FILTER XXX IS NOT NULL FILTER city == "ATLANTA" "

1 Like
8

I added the backtick suggestion and updated the object property descriptions and it runs better - Thanks for the help.

9

Glad to hear it! After this discussion we noticed one piece of our compiled prompt was contributing to this backtick issue so we pushed a fix EOD yesterday. You will hopefully notice less issues in a few days.