Running PySpark Tests in Code Workspaces

Ask the Community
, , ,
1

What’s the preferred pattern to run Spark DataFrame based tests using Code-Workspaces please?

We have a few code respositories that run tests using the pattern described in the Docs DataFrame Unit Test Example, e.g.

def test_dataframe(spark_session):
    df = spark_session.createDataFrame([['a', 1], ['b', 2]], ['letter', 'number'])
    assert df.schema.names == ['letter', 'number']

But this doesn’t work in Code-Workspaces, the tests fail because there is no spark_session available.

I can find the existing fixture for this which is at site-packages/foundry_pyls/fixtures.py (code not posted because of copyright notice) and add that into my own conftest.py file as a PyTest Fixture, but it doesn’t work.

It additionally requires installing the ‘Pyspark’ Python package, but:

  1. That doesn’t seem to work, even with Pyspark installed it can’t find Spark
  2. Adding Pyspark directly like this seems to conflict with the Foundry Transforms imports, causing other issues.

How can we run Spark DataFrame tests in Code Workspaces please? Currently we’re stuck using Code Repositories for these parts.