Hi all, any advice on approaches to make and locate documentation for pipelines?
What approaches / tools seem best to document pipelines where documentation will then be available to key clients and Palantir staff?
Asking especially since pipelines are split over multiple projects, so it doesn’t seem fitting to just keep a collection of Notepads just in one particular project.
Documentation on pipelines needs to be mostly for those clients and Palantir members maintaining the pipelines. Correct me if I’m wrong but Foundry Custom Documentation wouldn’t be a good fit since it’s aimed at all users?
Good question. Our public docs share some useful points to keep in mind when writing pipeline docs, which I’ll link to below:
A key point that these docs share is the importance of keeping pipeline docs close to the pipeline it describes, meaning that the custom docs repo likely isn’t the best place to store these since that would place them away from the pipeline project. From the docs:
An example of a good location is to keep the documentation close to the pipeline in Foundry, such as in a top-level "documentation" folder of the Project where the key outputs of the pipeline live.
An additional docs callout in our docs on recommended pipeline project and team structure adds these tips:
If you want to avoid using Notepad, though it is the recommend tool for thorough process instructions/escalation plans, I’d suggest adding a documentation folder to your code repository to record these things in Markdown and linking out to additional resources if needed.
Let me know if I’m missing some context here, but hopefully this helps.
Many thanks, the challenge is that we have 50-60 pipeline projects and many SOP docs and pipeline diagrams / descriptions that need to span many projects. If the documentation is kept close to the pipeline it’ll be very fragmented and potentially duplicative.
The resources linked are very useful as references though, thank you.
It won’t answer the question of “where to store it”, but if you can group the pipelines that belongs together, you can maybe check out Solution Designer, as a way to build an abstracted version of your pipeline.
You can link nodes on the graph to existing resources on your Foundry enrollment (Notepad, code repo, pipeline builder, datasets, …) so that you have an overview of your pipeline and the documentation available.
Many thanks, this looks really nice, if we can make the diagram fragmented and modular enough for the hundreds of pipeline steps this should be doable. Likely we’ll need to consolidate steps a bit to be easier on the eye.
