[Polars incremental] Error when reading previous output with set mode "replace"

Ask the Community
1

Hi,

When I use .set_mode(“replace”) before reading the previous output, I have an error FoundryDataSidecar:CannotDownloadSchemalessDatasetAsTabular, even if I specified a FoundryFieldSchema.

The current output is not empty and has the same schema I defined.

Note that when I use the .set_mode(“replace”) AFTER reading the previous output it works..

def compute(
    ctx,
    snapshot_scores: IncrementalLightweightInput,
    processed_scores: IncrementalLightweightOutput,
):
    FINAL_SCHEMA = FoundryFieldSchema(
        [
            FoundryFieldColumn("key", FoundryFieldType.STRING, nullable=True),
            FoundryFieldColumn(
                "value", FoundryFieldType.DECIMAL, nullable=True, precision=2, scale=0
            ),
            FoundryFieldColumn(
                "processed_at", FoundryFieldType.TIMESTAMP, nullable=True
            ),
        ]
    )
    processed_scores.set_mode("replace")
    # Read the input dataset
    input_df = snapshot_scores.polars(lazy=True)

    # Check if the transform is running incrementally
    if ctx.is_incremental:
        # Read the previous output to access already processed data
        final_df = processed_scores.polars(
            lazy=True, mode="previous", schema=FINAL_SCHEMA
        )

    else:
        # Non-incremental processing: process the entire input dataset
        final_df = input_df

    processed_scores.write_table(final_df)

According the documentation the behavior I try to reproduce is acceptable:

image
image841×781 65.8 KB

https://www.palantir.com/docs/foundry/transforms-python/incremental-usage#incremental-modes-of-inputs-and-outputs

Do you have any clue on how to fix the issue ?

Thanks.