In code repo, I use input_dataset.dataframe().columns to get all the columns. For example:
string_columns: list[Column] = [F.col(c).cast(T.StringType()) for c in df.columns]
df: DataFrame = df.withColumn('check_changes', F.concat_ws('_', *string_columns))
In Pipeline Builder, I need to carefully click “Add Item” for all ~100 columns, in the correct order, in the concatenate string transform. Please enable an option to either select all, or preferably a way to paste a list of columns, similar to how the Select transform allows for pasting a list of columns. (This would also help with the pain of manually dragging the handles to re-order 100 columns.)
