Python transforms for multiple datasets

Ask the Community
,
1

Hi! Maybe this is very basic but I’m having trouble finding a way to apply a same python transformation to multiple datasets.

I have a way to filter out my datasets by some words in the name of the dataset, the datasets are different between themselves (so making a union between them is a huge hasle), but my code would still work on all of them. However, as with python transforms you have to pass the RID each dataset I can’t find a way to apply my transformation to all of them.

Is there a way to pass a folder as an input and go over all the datasets inside it? Any ideas? Am I missing something here?

Thanks in advance.

2

It’s called „Transform generation“ in foundry world:

https://palantir.com/docs/foundry/transforms-python/transforms-pipelines/

2 Likes
3

for simpler use cases, you can encapsulate your transformation code into a function and call it from multiple compute
ex:

import common_transformation as ct
input =..
output=..
compute(input, output):
   out_df = ct.perform_whatever(input_df)
output.write(output_df)