We have been noticing new transformations that are being added when pipelines are installed via marketplace products. These nodes were not in the original pipeline. Moreover, some transforms are giving schema errors.
Hey @RajKarri, to confirm, are you saying the two âmarketplace transformsâ nodes werenât apart of the original pipeline builder that was saved + packaged? Whatâs going on in the Marketplace transform node?
This is literally the next node after each input node
Edit click on it shows something like this. We never added these things in the actual pipeline.
Ah I see, so itâs expected that there are added transforms (added by us) to make sure that the input datasets on install have the same schema as the inputs during packaging time. However, you shouldnât be getting schema errors. Can you share what error you got here?
Example for additional context:
You can package your pipeline with input columns A, B
, but install with input columns C, D, E
. During installation, you map C -> A
and D -> A
. We will rename columns C
and D
and drop column E
in the marketplace transform.
I guess thatâs the issue. Some of our input datasets are dynamic at times, which is causing the problem. Even though we never use those dynamic columns, âMarketplace Transformsâ is trying to ensure the columns remain consistent for mapping.