union_many is used to combine multiple dataframes.
One of the dataframe to be combined is missing a column, so the parameter how=“wide” is set.
In this case, the order of the columns is changed to alphabetical.
I want to keep the order of the columns in the input dataframe, so I am rearranging the order of the columns using dataframe.select again,
We think this is very inefficient.
We are trying to combine multiple dataframes with different columns while maintaining the order of the columns,
Is there any way to set the parameters of union_many or another method to join multiple dataframes with different columns while maintaining the column order?
It looks like the current implementation of union_many with the wide option always alphabetizes the columns.
Have you tried using built-in Spark unionByName with the allowMissingColumns argument set to True? I believe (though I haven’t tested it) that the built-in function uses some heuristics to preserve column order to the extent possible.
To my understanding, the query plan performance issues that made union_many essential for joining many dataframes in the past have been resolved in more recent Spark versions, so just calling unionByName in a for-loop should be fine now.
By setting allowMissingColumns=True in the unionByName argument,
I was able to unionize the dataframe with the missing columns without changing the order of columns in the original dataframe.
I would like to implement using unionByName in a for loop to unionize multiple dataframes.