Bypassing Cyclical Dependencies In Python Transforms

Ask the Community
1

I have:

  1. An output dataset which backs an Object Tyoe
  2. The Object Type has a materialized dataset
  3. I want to add the materialized dataset back as an input to my output dataset in (1).

This works fine on a branch, but fails with the error on main:

There are cyclical dependencies in the build graph.
Review your pipeline in the Data lineage graph, identify the area(s) where there is a cyclical dependency, and eliminate it.

Is there a way to suppress this check or bypass it since the logical flow of the pipeline is what I intend to do.

2

Hey @ramverma - there is no good way to bypass cyclical dependencies. We keep it in place so a dataset will not get into an error state that it then cannot escape.

If you can provide more detail on what you are attempting to accomplish there should be an alternative!

3

This may not work for your specific use case, but incremental transforms enable you to read data from a dataset that you want to write back to later on in the same transform.

Here’s a link to a previous community post regarding how to accomplish this: https://community.palantir.com/t/incremental-transforms-how-to-read-entire-output-dataset-at-start-of-execution/1860/2.

4

I had the same kind of issue when developing the creation of “event-objects” based in other events that then should be used to feed into the overall events again (as described in my second book).

I ended up in creation “event-proposal” Objects from my output and then within an action, triggered by an automated event, I shoveled the event-proposals into the the event table.

Not really a bypass on Dataset level but for smaller amounts of data this worked quite well. And the “shoveling” of the data is no dependency on dataset lineage level