Marketplace - How to split and package my workflow into Products?

[VincentF]

I have a project, or even multiples, with a lot of resources: Data sourcing, pipelines, Ontology (including Object Types, Actions, function, Automation , .. ), Applications, export pipelines, …
How should I package all my resources in marketplace ? Should I package everything in one massive product ? Or in multiple products ? Where should I do the splits ?

[VincentF]

There is no single official best practice or answer about “How to package you resources in Marketplace products”. However, below are elements to considers to make an informed choice.
Important note: If the workflow is simple, you should as well consider to package it in a single product. This is the simplest solution available.
Note: the word “workflow”, below, refers to a set of resources (data-sourcing, pipelines, objects, applications, models, etc.)

Considerations

Modularization and separation of concerns
Splitting a workflow by “semantics” or logic boundaries is often the simplest way to think about it.
e.g. “ontology”, “actions”, “objects”, “applications”, “pipelines” products

Avoid circular dependencies
Even if a resource depends on another, they can be in separate products. But if resources depends on each other, this can create circular dependencies which becomes problematic to resolve at installation time.
e.g. an object depends on a dataset (but not the opposite). An application depends on an object (but not the opposite). An Action depends on an object (but not the opposite). Those are monodirectional dependencies in classic cases and so can be packaged separately.
e.g. [Objects and Action] in one product and [Functions] in another product. As Function depends on Objects and Action depends on Functions, this generate a circular dependency.

Packaging/releasing frequency and needs
If part of the workflow are expected to evolve at different rhythms, it might make sense to package them separately.
e.g. data sourcing vs the rest. The data sourcing might evolve more slowly (new syncs added occasionally) vs the iterations on processing, ontology, applications, modeling, …
e.g. ontology vs. application. The application iterations (especially with Workshop) can be very quick, and require more frequent release than the Ontology requires.

Ownership
If part of the workflow will be owned by different teams, splitting the product might generate less friction to update and deploy the products.
e.g. if a team takes care of “data sourcing” and another of the “ontology and application”. Each team can have its own product, that it can update, maintain and deploy on their own schedules and with their own constraints

Permissions
Products are installed in their own project (at least in production mode). Hence if different parts of a workflow need to have different permissions setups, each parts of the workflow will need to be its own product, installed in its own project.
e.g. If “applications” needs to be access only by end users, and everything else will be accessible by the devs, then packaging the “application” in their own product, ultimately installed in its own project during deployment.

Technical Scale
Very big project might hit technical limits of Devops/Marketplace products. Even if those limits are high and increases over time, it might be relevant to split very large workflows in multiple products, to avoid hitting those limits and maintain a smooth workflow when packaging the resources.
Once a resource is part of a product, it is not possible to change “ownership” of the marketplace resource from one product to another easily, as of today. Hence, if the product is expected to grow significantly (>25%+) in size and you are already in reach of the limits, you should consider this technical split.
Note: Devops presents warning when a product start to be in reach of those limits, by showing you a “loading bar” of the capacity of the product vs actual capacity used.

Final notes

Note that a range of tooling helps you to manage multiple products together. Namely, “Linked Products” helps to coordinate and install multiple dependent products together in one go.
See https://www.palantir.com/docs/foundry/marketplace/linked-products

Example of a simple workflow split:

• Datasourcing and pipelining together - as the frequency of updates is expected to be less and the permission model is different than the rest.
• Ontology - as this is a reusable asset and can grow over time.
• Applications - combined or alone, as this requires different permissions (some user accessing App A won’t access App B), as well as the frequency of updates and team split (a team that build an app might not be the one responsible for the core ontology).