What are best practices for grouping multiple pipeline builder applications? We have about a dozen related pipelines in a given project and are currently just keeping them in a folder.
Is there a way similar to a code repository to have a single resource that helps group related pipeline builders in a single area/what are best practices beyond just keeping them all in a folder?
At the moment, we don’t have guidelines on how best to organize several interconnected pipelines. This is partially because organizational requirements vary, which may result in needing different projects for different pipeline stages. Additionally, some users may prefer to segment their pipelines by “pipeline stage” (raw/clean/transformed), whereas others may prefer to segment their pipelines by subject matter (airplanes, flights, etc).
As an alternative to a folder, you can save a Data Lineage graph that contains all the inputs and outputs from the pipelines in question. This won’t dynamically update as new datasets are added to the overall connected graph, but can provide a convenient way to zoom out and see the entire system end to end.