Hi everyone,
I have been looking over various posts on here, and I have found you can specify the response format as JSON when you make a request to to the a GPT model.
For example, this code works
# json mode with GPT
from palantir_models.models import OpenAiGptChatLanguageModel
from language_model_service_api.languagemodelservice_api_completion_v3 import GptChatCompletionRequest
from language_model_service_api.languagemodelservice_api import ChatMessage, ChatMessageRole
model = OpenAiGptChatLanguageModel.get("GPT_4o")
sys_prompt = "Your task is to come with a trivia question related to the user topic. Respond with JSON like {'question': trivia question, 'answer': trivia answer}"
request = GptChatCompletionRequest([ChatMessage(ChatMessageRole.SYSTEM,sys_prompt), ChatMessage(ChatMessageRole.USER, "The topic is science")], response_format={"type": "JSON_OBJECT"})
gpt_response = model.create_chat_completion(request)
print(f"This is request: {request}")
print()
print(f"This is the response: {gpt_response}")
Can you also specify JSON as the response format with llama models/generic completion language models? I have not found a solution to this yet. Here is what I’ve tried so far.
# structured output with llama
from palantir_models.models import GenericCompletionLanguageModel
from language_model_service_api.languagemodelservice_api_completion_v3 import GenericCompletionRequest
model = GenericCompletionLanguageModel.get("Llama_3_1_70b_Instruct")
prompt = "Your task is to come with a trivia question related to science. Respond with JSON like {'question': trivia question, 'answer': trivia answer}"
request = GenericCompletionRequest(prompt, response_format={"type": "JSON_OBJECT"})
llama_response = model.create_completion(request)
print(f"This is request: {request}")
print()
print(f"This is the response: {llama_response}")
The error message I get is
TypeError: init() got an unexpected keyword argument ‘response_format’
Additionally, I can’t find documentation anywhere about what parameters you can pass to the various models. I am aware of this documentation page (https://www.palantir.com/docs/foundry/integrate-models/language-models-transforms-inputs), but it does not specify what parameters like response_format and temperature that you can pass into requests.
Thanks,
Jenny
