I’m trying to use this code:
final_df = first_df.unionByName(second_df, allowMissingColumnsbool=True)
per https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.unionByName.html
Why do I get this error ?
transforms.external.systems._redact_credentials_in_output.TypeError: DataFrame.unionByName() got an unexpected keyword argument 'allowMissingColumnsbool'
The argument is only allowMissingColumns, the ‘bool’ after it in the docs just indicates the type, and then the default value.
final_df = first_df.unionByName(second_df, allowMissingColumns=True) should work.
Totally correct. Inattention mistake.
For some time it wasn’t available until spark upgraded, but indeed that’s not the issue here ! Thanks for the spotting !
It’s not always a wrong assumption to make. What’s still confusing me is the missing pyspark.testing in the code repos, which should be available from 3.5 onwards, but doesn’t appear to be in the Spark on our Foundry Instance.