Why can't I use allowMissingColumnsbool on unionByName

I’m trying to use this code:

final_df = first_df.unionByName(second_df, allowMissingColumnsbool=True)
per https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.unionByName.html

Why do I get this error ?
transforms.external.systems._redact_credentials_in_output.TypeError: DataFrame.unionByName() got an unexpected keyword argument 'allowMissingColumnsbool'

The argument is only allowMissingColumns, the ‘bool’ after it in the docs just indicates the type, and then the default value.

final_df = first_df.unionByName(second_df, allowMissingColumns=True) should work.

Totally correct. Inattention mistake.
For some time it wasn’t available until spark upgraded, but indeed that’s not the issue here ! Thanks for the spotting !

1 Like

It’s not always a wrong assumption to make. What’s still confusing me is the missing pyspark.testing in the code repos, which should be available from 3.5 onwards, but doesn’t appear to be in the Spark on our Foundry Instance.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.