Use LLM Transform on multiple (eg 2) input dataframes

Ask the Community
1

I have a straightforward use case - two input datasets - and I’d like to use an LLM to find matching rows from each, based on two columns.

It would be useful if “Use LLM” could take multiple inputs - so I could drag the two input datasets into the Use LLM transform, label one as ‘source’ and the other as ‘target’, and tell the LLM to find matching rows in ‘target’ for each row in ‘source’ based on free text in specific fields.

But, it looks like Use LLM is limited to one input dataframe, so I’m doing a full outer join and bringing that into Use LLM - but it seems clunky.. and also doesn’t work. Seems like Use LLM only handles one row at a time - so is unable to search the ‘target’ dataset for matching rows..

2

Hey @grumpbear you are correct in that currently the use LLM node in PIpeline Builder doesn’t support multiple inputs and that the current workaround is joining your datasets together. That being said we can track what you proposed in an feature request! Do you normally just use one dataset as a reference and the other dataset gets the actual LLM output or do you have more details on the most common use cases here?

3

thanks, most common use case is iterating over a source dataset and then searching for matching rows in a target dataset. In both cases we’d probably apply some filters and data prep to each dataset before asking the LLM to do the matching