Hello, we have an older repository with many PySpark transforms where we’re now trying to move to some Polars transforms, but we seem to be getting conflicts when we add a transform. I was wondering if anyone else has seen this, and knows how to resolve it please?
These are the key features of the repo:
In ‘gradle.properties’:
transformsLangPythonPluginVersion = 1.1088.0
and all upgrades have been applied. In ‘build.gradle’
apply plugin: 'com.palantir.transforms.lang.python-defaults'
// Apply the testing plugin
apply plugin: 'com.palantir.transforms.lang.pytest-defaults'
When I add a simple Polars transform (that works in Preview), then the checks fail with the following error:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':transforms-python:testVersions'.
> Execution failed with non-zero exit code: 1
[STDERR]
ERROR x The following packages are not available or the package requirements are not compatible with each other given the available versions.
Library
+-- pytest-xdist ==2.0.0 [conda]
+-- pandas ==3.0.0 [conda]
+-- pytest-xdist >=3.6.1 [conda]
Hawk version 0.469.0
The libraries added are:
In conda-versions.test.linux-64.lock we have:
...
palantir-spark-time=3.24.0=py_0 @ noarch
pandas=2.3.3=py312h23c847b_1 @ linux-64
pip=26.0.1=pyhc872135_0 @ noarch
pluggy=1.6.0=pyhf9edf01_1 @ noarch
polars-runtime-32=1.38.1=py310hffdcd12_0 @ linux-64
polars=1.38.1=pyh6a1acc5_0 @ noarch
...
pytest-forked=1.6.0=pyhd8ed1ab_1 @ noarch
pytest-html=1.22.0=py_0 @ noarch
pytest-metadata=2.0.4=pyhd8ed1ab_0 @ noarch
pytest-transforms=3.161.0=py_0 @ noarch
pytest-xdist=2.0.0=py_0 @ noarch
pytest=7.2.1=pyhd8ed1ab_0 @ noarch
python-dateutil=2.9.0post0=py312h06a4308_2 @ linux-64
python-gil=3.12.12=hd8ed1ab_2 @ noarch
python-tzdata=2025.3=pyhd3eb1b0_0 @ noarch
python=3.12.12=hd63d673_2_cpython @ linux-64
python_abi=3.12=8_cp312 @ noarch
...
and in ‘conda-versions.run.linux-64.lock’
palantir-spark-time=3.24.0=py_0 @ noarch
pandas=3.0.0=py312h8ecdadd_0 @ linux-64
pip=26.0.1=pyhc872135_0 @ noarch
polars-runtime-32=1.38.1=py310hffdcd12_0 @ linux-64
polars=1.38.1=pyh6a1acc5_0 @ noarch
...
pyspark-src=3.5.5.25=py_0 @ noarch
pyspark=3.5.5.25=py_0 @ noarch
python-dateutil=2.9.0post0=py312h06a4308_2 @ linux-64
python-gil=3.12.12=hd8ed1ab_2 @ noarch
python=3.12.12=hd63d673_2_cpython @ linux-64
python_abi=3.12=8_cp312 @ noarch
This repo doesn’t have a hawk.lock file.
I’m not really sure where the Pandas 3.0 dependency comes from, as we don’t have any pandas transforms, currently all PySpark, and the one Polars attempt.
Thinking that it was a conflict in the test libraries only, and not in the run, I removed the line apply plugin: 'com.palantir.transforms.lang.pytest-defaults' in ‘build.gradle’, and when I do that, the checks complete, but the build fails. It starts three times, but then ultimately fails with:
Unexpected infrastructure failure: the job has failed unexpectedly in the backing compute cluster without committing or aborting.
Reason: The following container(s) terminated unexpectedly: [container_name: user-container, state: ImagePullBackOff]
If I manually remove the entry pandas=3.0.0=py312h8ecdadd_0 @ linux-64 from run.lock, or if I add in its place pandas=2.3.3=py312h23c847b_1 @ linux-64, then checks succeed (with or without the tests enabled in the build.gradle file), and after the checks no new pandas version appears in the lock files, but the build again fails with the container error above. (I realise we’re not meant to by manually editing the .lock files, this was just a random attempt to resolve this issue.)