Is it possible to do a cosine similarity between two vectors (or dot product) in pipeline builder with two arrays?

Ask the Community
1

I have two vectors as columns and would like to compute cosine similarity between the two. There is an array cartersian product but not exactly what I need.

Perhaps there is a way to define a user function that can replicate a dot product (and then cosine similarity would be relatively straightforward) but I wasn’t able to figure out how to approach.

2 Likes
2

You should be able to write a UDF to do this - see docs https://www.palantir.com/docs/foundry/transforms-java/user-defined-functions/

3 Likes
3

Didn’t know you could write your own UDFs and publish - thank you!

4

Python functions have recently been enabled so you could also opt to do this in Python instead - the workflow is quite a bit easier too.

1 Like
5

Python functions was definitely the easiest solution given the new update! Easy to just spin up a repo (https://www.palantir.com/docs/foundry/code-repositories/python-functions/)

from functions.api import Array, Float, function
from numpy import dot
from numpy.linalg import norm


#
# This code defines a function called vector_cosine_similarity that takes two input arguments
# event_embedding and query_embedding, both of which are arrays of floating-point numbers.
# The function calculates the cosine similarity between these two input vectors by computing
# the dot product of the vectors and dividing it by the product of their norms (magnitudes).
# The cosine similarity is a measure of similarity between two vectors, with a value ranging
# from -1 (completely dissimilar) to 1 (identical).
#
@function
def vector_cosine_similarity(event_embedding: Array[Float], query_embedding: Array[Float]) -> Float:
    return dot(event_embedding, query_embedding) / (norm(event_embedding) * norm(query_embedding))

thanks for the help, both!

1 Like