Generate several builds from a single transform in Python

Ask the Community
1

Hello Community Team,

I was wondering if there is a way to generate several builds from one transform in Python ? ( it means that a single transformation will have for example 10 datasets in output, and each dataset in the output will have his own build and his own spark configurations ( so if we have in the profile 10 executors, each build will have 10 executors ) , so it’s not a multi output transformations because in that case the datasets have the same build and share the ressources defined in the profile, I know we can do that in MypipelineDefiner in java, but i don’t know if we can do something similar in Python ?

Best Regards,
Soufiane

2

Hi @Soufiane,

You can achieve this with a Transform Generator which uses a loop to generate individual transforms: https://www.palantir.com/docs/foundry/transforms-python/transforms-pipelines/#transform-generation.