Hi
Context:
When resources are shared in producer-consumer workflows using ontology projects , resource-level security markings can be introduced by the producer and propagated downstream to child resources. When a producer makes changes without prior notice—such as adding or swapping markings—, these changes can inadvertently impact consumers. Specifically, if a consumer receives a newly propagated marking without warning, it may cause unexpected access restrictions or application blackouts. This issue becomes particularly complex to solved when incremental transforms are involved, as the resulting disruption can be prolonged and difficult to trace. While security enforcement via marking propagation in incrementals is well-thought through, consumers are often unaware that a single marking change in one transaction can impact the visibility of an entire dataset.
At a more abstract level, avoiding such disruptions relies on two things:
- Tools to understand the downstream impact of marking changes (some of which exist today - (Producer side: Data Lineage • Understand permissions • See the impact of marking changes • Palantir, or on the consumer side Management • Manage Project constraints • Palantir),
- Proactive communication between producers and consumer**, which is often inconsistent due to varying organizational structures or differing levels of software development rigor.
Feature Request:
Either:
- Enhance the Expectations API to include functionality that allows consumers to prevent or safeguard against unexpected propagation of markings.
from transforms import expectations as E
@transform(
out=Output("....",
checks=[
Check(E.no_marking_expected, "No markings expected", on_error="FAIL"),
or
Check(E.marking_expected("marking_rid1"), "only this marking expected", on_error="FAIL"),
],
),
)
....
or, alternatively,
- Enhance the Project constraints functionality ( Management • Manage Project constraints • Palantir): This can also be considered as an alternative to enhance the expectations API, however, it lacks the functionality that no markings should be expected in the project at all.
This feature would allow to establish a first line of defense producer-consumer workflows where data is shared.
Problem it Solves:
This enhancement would help consumers mitigate data access issues resulting from uncommunicated upstream marking changes, thereby improving system resilience and reducing resolution time. When applied at integration boundaries, it would ensure that downstream systems remain functional—potentially operating with outdated data, but not experiencing full outages.
Considerations
I understand the other side of the coin—where a producer may have inadvertently left data unprotected and needs to urgently propagate a marking to downstream resources. Introducing these checks would allow consumers to continue accessing previously available sensitive data until they choose to update their logic, though no newly marked data would be ingested in the meantime. While similar protections can already be implemented using the existing API ( List Markings Of Resource • API Reference • Palantir), enhancing the Expectations API would make this workflow more intuitive and convenient to adopt.
