Spark 3.5.1 Upgrade
The version of Spark used in Foundry transforms is being upgraded from 3.4.1 to 3.5.1. This brings a plethora of bug fixes, performance improvements, and new features that can be viewed on the release page. No action is required to enable this upgrade. The transforms runtime versions are managed automatically in the background through adjudication, as per the transforms versions docs.
The Spark versioning policy disallows the introduction of API and behavior changes between minor versions. Any changes are documented in the migration guide, which has been tested by Palantir to ensure the user experience remains constant.
It is possible for upgrades to affect the performance of pipelines and introduce new bugs. For high-risk jobs, use a temporary pin to control the upgrade rollout until you have confirmed the upgrade to be issue-free for your workflows.