Aborting incremental media set transform

Ask the Community
,
1

If I’m writing a regular dataset incremental transform in Python, I can abort the output when relevant to prevent triggering downstream updates:

from transforms.api import transform, incremental, Input, Output


@incremental()
@transform(
    my_input=Input("ri.foundry.main.dataset.12345"),
    my_output=Output("ri.foundry.main.dataset.67890")
)
def compute(my_input, my_output):
    # [...]

    if no_new_output:
        my_output.abort()
        return

    # [...]

However, when writing media set transforms, this causes an error: AttributeError: 'MediaSetOutputParam' object has no attribute 'abort'.

from transforms.api import transform, incremental
from transforms.mediasets import MediaSetInput, MediaSetOutput


@incremental(v2_semantics=True)
@transform(
    media_in=MediaSetInput("ri.mio.main.media-set.12345"),
    media_out=MediaSetOutput("ri.mio.main.media-set.67890"),
)
def translate_images(media_in, media_out):
    # [...]

    if no_new_output:
        media_out.abort()
        return

    # [...]

How can we write incremental media pipelines that can abort transactions?

2

Call abort_job() on the TransformContext object:

from transforms.api import transform, incremental
from transforms.mediasets import MediaSetInput, MediaSetOutput


@incremental(v2_semantics=True)
@transform(
    media_in=MediaSetInput("ri.mio.main.media-set.12345"),
    media_out=MediaSetOutput("ri.mio.main.media-set.67890"),
)
def translate_images(ctx, media_in, media_out):
    # [...]

    if no_new_output:
        ctx.abort_job()
        return

    # [...]

Docs:

  • https://www.palantir.com/docs/foundry/transforms-python/incremental-media-sets#abort-incremental-transforms
  • https://www.palantir.com/docs/foundry/transforms-python/abort-transactions#how-do-aborted-transactions-relate-to-incremental-transactions
2 Likes