In Pipeline Builder, custom transform paths cannot pass a column list argument into a “Select columns” transform. In the screenshot below, I was able to pass the Column Argument, but it was not possible to pass the Column List Argument instead. I am trying to create a reusable transform to replace my select, filter, and clean steps that I need to manually create for many input SAP datasets.
Hi Joel, I ran into the same issue recently.
From what I observed, the Select Columns transform accepts only individual Column<Any> arguments. Although a Custom Transform Path lets you define a List<Column<Any>> argument, the Select Columns node doesn’t seem to bind list-type arguments—only single-column arguments.
My use case is similar to yours: I wanted to build a reusable transform for Select → Filter → Clean across multiple SAP datasets without duplicating the same logic in every pipeline.
The workaround that worked best for me was moving this logic into a Python (PySpark) transform. It allows you to parameterize the list of columns, filters, and cleaning logic in one reusable function, and it’s much easier to test and maintain.
def select_filter_clean(df, columns, filter_expr=None, clean_cols=None):
df = df.select(*columns)
if filter_expr:
df = df.filter(filter_expr)
if clean_cols:
for c in clean_cols:
df = df.withColumn(c, clean_string_udf(col(c)))
return df
I’m also curious if anyone has found a Pipeline Builder–only solution or if this is simply a current limitation of the Select Columns transform.