Issue / Behavior Description: We have identified a edge-case behavior with Foundry Dataset Schedules that, while technically working as designed, can easily lead to silent data freshness issues and confusion for users.
Scenario / Example:
We have a downstream Dataset C defined as the simple union of two upstream Datasets, A and B.
Dataset C has a schedule set to trigger when BOTH parent datasets (A and B) complete their build.
Under normal operations, both A and B complete their builds around 7:00 AM daily, which then triggers C.
The Problem: If a user manually triggers or rebuilds Dataset B during the day (e.g., at 4:00 PM):
The schedule system records Dataset B’s build as complete/up-to-date.
The schedule then waits for Dataset A’s next build.
The next morning at 7:01 AM, Dataset A finishes its build.
Dataset C immediately builds because both conditions are now fulfilled (A built at 7:01 AM today, and B built at 4:00 PM yesterday).
Impact: Dataset C gets built using today’s data from A combined with yesterday’s data from B, completely skipping the fresh 7:00 AM build of B. This creates an unexpected temporal shift/dephase between datasets that is very hard to detect and audit.
Questions & Feedback:
What is the recommended workaround / best practice in Data Connection / Data Lineage to enforce strict timestamp/synchronization constraints for multi-parent schedules?
Is there any planned enhancement from Palantir to add staleness warnings, maximum lag windows, or relative time constraints on build triggers to help users prevent and detect these silent desynchronization issues?
I recently wrote a fairly lengthy post on a similar kind of setup: https://www.linkedin.com/pulse/quorum-gate-scheduling-pipelines-completeness-time-louis-gruand-fkmbe/
You’re right that the schedule is working as designed. The catch is that an “all parents built” trigger only checks that each parent built, not when or whether they’re aligned in time, so it happily fires on a stale B. It’s in-spec but out of bounds for what you actually want.
My first question back to you: do you expect this to happen often?
If it’s rare, I’d probably just live with it and rely on manual awareness. If it happens with any regularity and the temporal alignment matters, I’d encapsulate the trigger logic in an intermediate “gate” dataset rather than triggering C directly off A and B.
The idea: instead of C being scheduled on “A and B both built,” you build an intermediate dataset that captures the freshness logic the scheduler can’t express. For example, compare the max date_added (or your equivalent watermark) across A and B, and only emit a row / mark the dataset updated when they actually agree, e.g. both reflect the same day’s build. C then schedules off that intermediate gate. That way C only builds when A and B are genuinely in sync, and a mid-day rebuild of B alone won’t slip through.
So my recommendation: if this is occasional and the logic is encapsulable, build the intermediate dataset. It gives you a place to put the timestamp/synchronization constraint the native schedule doesn’t offer.
On your second question (planned staleness warnings / max lag windows / relative-time triggers from Palantir), I can’t speak to the roadmap, so I’ll leave that one for someone from Palantir to answer.
What is the recommended workaround / best practice in Data Connection / Data Lineage to enforce strict timestamp/synchronization constraints for multi-parent schedules?
Some advice on options to consider for workarounds to be resilient to the edge cases you describe based on my personal experience:
I would generally recommend in this setup that you have a schedule X that builds A & B, and you set the trigger condition for schedule Y that builds C to be “schedule X built successfully”. This deals with both the time synchronization and guarantees you cannot get out of sync. (Edge case: If there are multiple such schedules that build A & B you can add them as ORs, or refactor A & B out to be their own shared unit of schedule between the other schedules)
Fwiw In practice I would say ~90% of my (non-ingest) schedule setups are based on “schedule X built successfully” conditions
If you want to stick to data updated conditions and be resilient to the above edge cases, I believe you should be able to use a setup such as ((A OR 15 minutes) AND (B OR 15 minutes)) AND (A OR B) as below using Advanced configuration to write the nested ORs and ANDs. This means if A updates, the schedule will wait up to N = 15 minutes (see complexity below around aligning schedule timings with the Run every N minutes and Starting at M past the hour) for B to update, and if it does not, it will build the schedule anyway. In your expected case, A and B update within N minutes of each other so the schedule runs as normal. In the edge case of someone only updating one, the schedule will build after <=N minutes, and then when you reach 7am again, the schedule will still be in a healthy state. The final condition of AND (A OR B)stops the schedule no-op triggering every N minutes.
The complexity here is that you need to align the Run every N minutes and Starting at M past the hour to align with when A and B are supposed to update, as the condition is not “Wait 15 minutes after A updates” it is “Wait until X:15am/pm”. So if your schedule runs on the hour, maybe you would set N=60 and M to be at 45 minutes past the hour to “reset the schedule to healthy” 15 mins before it is supposed to run again". Although I think there are lots of ways to think about what to set N and M here.
(Oh and if you keep your schedule as is, and the parents ever get out of sync again and you want to fix this, you can reset the schedule state by Pausing and Unpausing the schedule)
I agree with @max 's reply about using a “schedule run succeeded trigger” to trigger your schedule if A and B can (and are expected to) be built together.
Also in regards to your second question about staleness warnings and max lag windows, we offer a product called Monitoring Views (link to documentation) which offers a variety of different alerts you can set up based on certain rules to proactively flag such desynchronization issues. Specifically, for Schedules, we offer two rules:
fire alerts when a schedule fails consecutively more than X times
fire alerts when a schedule’s run duration exceeds a certain amount of time
In combination with @max and @louis-sibyl 's suggestions here, you could leverage these alerting rules (and others) to be notified if your pipeline is going out of sync or is lagging behind.
