Pipeline installations via markeplace adding new transformations

We have been noticing new transformations that are being added when pipelines are installed via marketplace products. These nodes were not in the original pipeline. Moreover, some transforms are giving schema errors.

Hey @RajKarri, to confirm, are you saying the two “marketplace transforms” nodes weren’t apart of the original pipeline builder that was saved + packaged? What’s going on in the Marketplace transform node?

This is literally the next node after each input node

Edit click on it shows something like this. We never added these things in the actual pipeline.

Ah I see, so it’s expected that there are added transforms (added by us) to make sure that the input datasets on install have the same schema as the inputs during packaging time. However, you shouldn’t be getting schema errors. Can you share what error you got here?

Example for additional context:
You can package your pipeline with input columns A, B , but install with input columns C, D, E . During installation, you map C -> A and D -> A . We will rename columns C and D and drop column E in the marketplace transform.

1 Like

I guess that’s the issue. Some of our input datasets are dynamic at times, which is causing the problem. Even though we never use those dynamic columns, ‘Marketplace Transforms’ is trying to ensure the columns remain consistent for mapping.