Why do I get this error ? transforms.external.systems._redact_credentials_in_output.TypeError: DataFrame.unionByName() got an unexpected keyword argument 'allowMissingColumnsbool'
Totally correct. Inattention mistake.
For some time it wasn’t available until spark upgraded, but indeed that’s not the issue here ! Thanks for the spotting !
It’s not always a wrong assumption to make. What’s still confusing me is the missing pyspark.testing in the code repos, which should be available from 3.5 onwards, but doesn’t appear to be in the Spark on our Foundry Instance.