JSON mode with Llama LLM

Hi everyone,

I have been looking over various posts on here, and I have found you can specify the response format as JSON when you make a request to to the a GPT model.
For example, this code works

# json mode with GPT
from palantir_models.models import OpenAiGptChatLanguageModel
from language_model_service_api.languagemodelservice_api_completion_v3 import GptChatCompletionRequest
from language_model_service_api.languagemodelservice_api import ChatMessage, ChatMessageRole
model = OpenAiGptChatLanguageModel.get("GPT_4o")
sys_prompt = "Your task is to come with a trivia question related to the user topic. Respond with JSON like {'question': trivia question, 'answer': trivia answer}"
request = GptChatCompletionRequest([ChatMessage(ChatMessageRole.SYSTEM,sys_prompt), ChatMessage(ChatMessageRole.USER, "The topic is science")], response_format={"type": "JSON_OBJECT"})
gpt_response = model.create_chat_completion(request)
print(f"This is request: {request}")
print()
print(f"This is the response: {gpt_response}")

Can you also specify JSON as the response format with llama models/generic completion language models? I have not found a solution to this yet. Here is what I’ve tried so far.

# structured output with llama
from palantir_models.models import GenericCompletionLanguageModel
from language_model_service_api.languagemodelservice_api_completion_v3 import GenericCompletionRequest

model = GenericCompletionLanguageModel.get("Llama_3_1_70b_Instruct")
prompt = "Your task is to come with a trivia question related to science. Respond with JSON like {'question': trivia question, 'answer': trivia answer}"
request = GenericCompletionRequest(prompt, response_format={"type": "JSON_OBJECT"})
llama_response = model.create_completion(request)
print(f"This is request: {request}")
print()
print(f"This is the response: {llama_response}")

The error message I get is

TypeError: init() got an unexpected keyword argument ‘response_format’

Additionally, I can’t find documentation anywhere about what parameters you can pass to the various models. I am aware of this documentation page (https://www.palantir.com/docs/foundry/integrate-models/language-models-transforms-inputs), but it does not specify what parameters like response_format and temperature that you can pass into requests.

Thanks,
Jenny

Hi Jenny :waving_hand:

The documentation you pointed out is a good source for parameters, also the model provider documentations which can be found here are a good place to look.

Also this sounds like a problem that AIP Assist can help with if you haven’t tried to use it yet. Feel free to reach out here if you are still having problems.