If I have one dataset that contains values that I want to look up (one cell per value) and another dataset that has definitions for those values (stored as key value pairs in a map), what would be the best way to
a. For every value in every row in dataset A, look up its definition in dataset B
b. Write the definition to a new column in dataset A
c. In the most time-efficient way possible
Any help is appreciated!
Matching values in one column of one dataset with the values in a different column of a dataset can be done through a Join operation. For Foundry datasets, Contour provides the ability for users to implement joins via a Join board. You can match the values across columns between 2 datasets or analysis paths, combine columns to create a single output, and then use other subsequent boards to manipulate or choose the subset of columns you want to keep. Check out this documentation and example of joining 2 datasets in Contour here: https://www.palantir.com/docs/foundry/contour/boards-join ! Contour is going to be the quickest way in Foundry to get this answer.
If you’re also looking to integrate this logic within a data pipeline, check out these examples:
- Joining datasets via Pipeline Builder: https://www.palantir.com/docs/foundry/pipeline-builder/transforms-join-data/
- Joining datasets within a pyspark transform: https://www.palantir.com/docs/foundry/transforms-python/pyspark-joins/