Hey! I have two datasets with ~40 overlapping columns that I want to join. If value exists in the overlapping column in dataset A i want to keep it, else use the one from dataset B. I can join them using a normal left join, add prefix and then set up a coalesce for each column but have to do this manually as I’m not able to do a pairwise “apply to multiple columns”. Is there an easier way?
Hey I think your current approach is the best way to do this today
In repository you could use a list comprehension. In pipeline, could you unpivot by column name so that you have two datasets, index, column_name, value. You could then join on index and column_name, to get a dataset with index, column_name, value, value_2, coalesce value, value_2 to get combined_value. Finally you can unpivot back out on the first value in the column names.