Under what conditions can a pipeline builder generated transform reuse an already live spark module?
We re-use a live spark module for previewing pipelines but actual pipeline builds request a spark module from Spark Module Manager in the exact same way as code transforms.
Spark Module Manager will give code transforms (different for SQL transforms) a used spark module, if there is an available module with the same spark profile and the transform comes from the same repository. Is being in the same pipeline builder file equivalent to being in the same repository?
Transforms Worker uses a number of criteria to determine if a module can be reused which include things like markings, eddie version and a few other things. We’re looking to make some improvements here in the coming weeks which should bump module reuse for Pipeline Builder jobs, I’ll check back in when we have more of an update