Build a pipeline "if this upstream dataset updates" but "only on monday"

Ask the Community
1

I have 2 pipelines:

  • [A] One upstream that updates once per day
  • [B] One downstream that needs to update only once per week

I want the downstream one to trigger only after the upstream one built. Otherwise, that would introduce a 24h latency if not worst, in the processing.

So the condition for building [B] becomes “Build B only if we are on monday and [A] built [today]”.

How can I encode that in a schedule ?

2

Hey @VincentF, in order to achieve this behaviour, you can use an AND trigger that combines two or more triggers and only runs when both are satisfied.

Using your specific example above, you could have something similar to the below screenshot setup

schedule
schedule359×339 15.2 KB

Let me know if that helps!

3

I don’t think this will work.

If on Monday, at 12:00PM the upstream pipeline [A] hasn’t yet run “on monday”, then this condition will still trigger.

Given Pipeline [A] builds everyday, the first condition will always be met when the second condition kicks in.

What I’m looking for is rather a “When [A] builds [today] then [run B]”

4

I see what you mean now. There can be two ways to configure this:

  1. On [A], have two schedules, one that builds every Monday (S1) and another that builds all other days of the week (S2).

  2. On [B] - configure the schedule to run whenever S1 has run successfully using an Event Trigger.

If you do not want to have two schedules on [A], another option to consider would be:

  1. On [A] - leave the schedule as-is.
  2. On [B]:
    a. Configure the schedule on [B] to run when the schedule on [A] succeeds.
    b. Update the logic backing [B] to abort the job when the current day of the week is not Monday.

Let me know if this helps. Unfortunately, with the currently supported triggers, there is no first-class way to specify that [B] should run only on a successful run of [A] on Mondays.

2 Likes