Hi,
Im not a seasoned developer, I know my way around Python. Now with the help of LLM’s I have been venturing into writing code in other languages.
I apologize if its a stupid question Im asking.
I have been trying to write a function that will use Claude models via bedrock from my account. The goal is to register the model and use it in AIP logic.
Here is my code
import { Function, Integer } from "@foundry/functions-api";
import { ExternalSystems, UserFacingError } from "@foundry/functions-api";
import { BedrockRuntime } from "@foundry/external-systems/sources";
import { Sources } from "@foundry/external-systems";
import { ChatCompletion } from "@palantir/languagemodelservice/contracts";
import {
FunctionsGenericChatCompletionRequestMessages,
GenericCompletionParams,
FunctionsGenericChatCompletionResponse,
} from "@palantir/languagemodelservice/api";
import {
BedrockRuntimeClient,
ConverseCommand,
ConversationRole,
Message,
} from "@aws-sdk/client-bedrock-runtime";
// Import the Node.js crypto module
import { randomBytes } from 'crypto';
import { Sha256 } from "@aws-crypto/sha256-js";
import { HttpRequest } from "@aws-sdk/protocol-http";
import { SignatureV4 } from "@aws-sdk/signature-v4";
// Add a polyfill for crypto.getRandomValues
const cryptoPolyfill = {
getRandomValues: function(buffer: Uint8Array): Uint8Array {
const bytes = randomBytes(buffer.length);
buffer.set(bytes);
return buffer;
}
};
// Set global crypto if it doesn't exist
if (typeof global.crypto === 'undefined') {
(global as any).crypto = cryptoPolyfill;
}
export class BedrockLLMFunctions {
@ExternalSystems({ sources: [BedrockRuntime] })
@ChatCompletion()
public async wueryBedrock(
messages: FunctionsGenericChatCompletionRequestMessages,
params?: GenericCompletionParams
): Promise<FunctionsGenericChatCompletionResponse> {
console.log("Starting queryBedrock function");
try {
const SECRETKEY = Sources.BedrockRuntime.getSecret("additionalSecretSECRETKEY");
const ACCESSKEY = Sources.BedrockRuntime.getSecret("additionalSecretACCESSKEY");
const region = "us-east-1";
console.log("Retrieved secrets for Bedrock Runtime");
// Create a custom signer
const signer = new SignatureV4({
credentials: {
accessKeyId: ACCESSKEY,
secretAccessKey: SECRETKEY,
},
region: region,
service: "bedrock-runtime",
sha256: Sha256,
});
// Initialize the client with the custom signer
const client = new BedrockRuntimeClient({
region: region,
credentials: {
accessKeyId: ACCESSKEY,
secretAccessKey: SECRETKEY,
},
signer,
maxAttempts: 3,
});
console.log("Initialized BedrockRuntimeClient");
const conversation: Message[] = messages.map((m) => ({
role: m.role.toLowerCase() as ConversationRole,
content: [{ text: m.content }],
}));
const command = new ConverseCommand({
modelId: "anthropic.claude-3-haiku-20240307-v1:0",
messages: conversation,
inferenceConfig: {
maxTokens: 512,
temperature: 0.5,
topP: 0.9,
},
});
console.log("Sending command to Bedrock");
const response = await client.send(command);
if (!response.output?.message?.content?.[0]?.text) {
throw new UserFacingError("Invalid response format from Bedrock");
}
const completionText: string = response.output.message.content[0].text;
return {
completion: completionText,
tokenUsage: {
promptTokens: response.usage?.inputTokens ?? 0,
completionTokens: response.usage?.outputTokens ?? 0,
maxTokens: 512,
},
};
} catch (error: unknown) {
console.error("Error during queryBedrock execution:", error);
const errorMessage = error instanceof Error
? `Error calling Bedrock: ${error.message}\nStack: ${error.stack}`
: "An unknown error occurred while calling Bedrock";
return Promise.reject(new UserFacingError(errorMessage));
}
}
}
When I publish the code and run the code with input
{
"messages": [
{
"role": "user",
"content": "why is the sky blue"
},
{
"role": "system",
"content": "why is the sky blue"
}
],
"params": {
"temperature": 1,
"maxTokens": 1234
}
}
I get error
LOG [2024-11-21T16:03:18.789Z] Starting queryBedrock function
LOG [2024-11-21T16:03:18.791Z] Retrieved secrets for Bedrock Runtime
LOG [2024-11-21T16:03:18.8Z] Initialized BedrockRuntimeClient
LOG [2024-11-21T16:03:18.8Z] Sending command to Bedrock
ERROR [2024-11-21T16:03:18.816Z] Error during queryBedrock execution: {}
UserFacingError: Error calling Bedrock: Secure random number generation is not supported by this browser.
Use Chrome, Firefox or Internet Explorer 11
Stack: Error: Secure random number generation is not supported by this browser.
Use Chrome, Firefox or Internet Explorer 11
at s.getRandomValues.t.exports (UserCode:2:886319)
at Object.getRandomValues (UserCode:2:942920)
at de (UserCode:2:38444)
at me (UserCode:2:38634)
at UserCode:2:88314
at async UserCode:2:89980
at async p.wueryBedrock (UserCode:2:943847)
at async Ne (FunctionsIsolateRuntimePackage:2:1080191)
at async ye.executeFunction (FunctionsIsolateRuntimePackage:2:1080698)
at async userFunction (FunctionsInitialization:10:43)
I have tried several prompting strategies with several LLM’s which is how I got “polyfill” to bypass the crypto error, then now Im in this place where random generation is not supported by this browser.
I tried to reach to folks who work on AWS SDK and they say, its issue with the runtime not being compatible with Node.js.
Anyone tried using AWS SDK in a typescript function before ? any recommendation on how to make it work ?