Recent Datatype Changes using lightweight

Ask the Community
1

We have observed a recent change in Foundry that affects how data is read and written when using transforms.api.LightweightInput. It appears that the automatic datatype harmonization previously applied in Lightweight transforms has been disabled or changed.

As a result, this can cause compatibility issues between “Foundry datatypes” and “Other datatypes”, especially when certain types are not supported on one side or were previously auto‑cast.

Observed Issues

  1. MapType columns with Polars input

Reading a dataset that contains a MapType column via input.polars() now results in a build-time error:

ValueError: Discovered an input column 'MAP' in the Foundry schema which is a Map type.
Polars does not support Map types. Please use another runtime, convert this column to Struct if possible,
or remove this column from the schema of the input dataset.

Previously, this MapType was automatically cast to a pl.Struct, so no manual intervention was required.

  1. Non-Foundry datatypes (e.g. pl.Enum)

When creating a non‑Foundry datatype in a Lightweight transform (e.g. a pl.Enum), the Parquet output correctly preserves this type.

  • When reading the same data via spark_input.dataframe(), the datatype is automatically cast (e.g. to StringType).
  • In the past, the same automatic casting also happened when reading via input.polars().

With the current behavior, reading via input.polars() now fails with:

polars.exceptions.SchemaError: data type mismatch for column CATEGORY:
incoming: Enum([...]) != target: String

Reading workaround

Polars’ own schema inference still works correctly, so reading the input manually bypasses the issue:


pl.scan_parquet(input_dataset.path() + "/**/*.parquet")

Writing workaround

Before calling out.write_table(df), datatypes need to be manually harmonized to valid Foundry datatypes to avoid downstream schema mismatches.

2

Hi @Solv -

Thanks for reporting the issues above. The changes you are seeing in Lightweight behavior are due to a recent improvement that allowed Python Lightweight reads to apply the Foundry schema of your dataset on read, which allows new workflows like schema evolution in incremental transforms that weren’t possible before. This is also crucial for interoperability of datasets within the platform - enabling dataset reads and writes between Spark, Polars, and other runtimes within the platform.

You’re correct that Polars will throw when its internal type system differs from the Foundry Schema type system, and I can confirm your workaround is valid for applying your own Polars schema as an “override” for the read. We’re tracking this issue and will shortly release the ability to pass these Polars schemas directly to the input.polars() call to allow you to remediate these schema issues on a case by case basis. I’ll follow up when this ships!