When I create the results field inside the tests array, it is being output as a String instead of an Array<Struct>.
Is nested Array<Struct<Array<Struct>>> supported in Pipeline Builder? If so, what is the recommended approach to preserve the nested array instead of having it serialized as a string?
Any guidance or workarounds would be greatly appreciated. Thanks!
This is possible in Pipeline Builder, you just need to make sure every struct in tests has the same schema, and same for every struct in results . If you set tests to Array<String> and then try to add a non-String element Pipeline Builder will error.
Example:
Creating the top-level array/setting up the schema:
The behavior you are seeing, where results is being output as a String instead of an Array, is consistent with a limitation of Pipeline Builder’s no-code interface when handling deeply nested complex types.
What is happening
Pipeline Builder supports Array data types and allows access to nested struct fields using dot notation (for example, tests.someField). However, when a struct field itself contains an Array, resulting in a structure such as Array<Struct<Array>>, the schema construction interface cannot fully represent the nested array type. In these cases, the inner Array is often serialized as a JSON string, which is the behavior you are observing.
While this limitation is not explicitly documented, it aligns with the documented behavior around converting struct columns to JSON strings and with existing notes that complex type casting (including structs, arrays, maps, and conversions between them and strings) may behave differently than expected.
Recommended approach: Python Code Repository
The most reliable way to create and preserve a schema such as Array<Struct<Array>> is to implement the transformation in a Python Code Repository, where the schema can be defined explicitly using PySpark’s StructType and ArrayType.
from transforms.api import transform_df, Input, Output
from pyspark.sql import types as T
SCHEMA = T.StructType([
T.StructField(“sample_id”, T.StringType()),
T.StructField(“tests”, T.ArrayType(
T.StructType([
T.StructField(“test_name”, T.StringType()),
T.StructField(“results”, T.ArrayType(
T.StructType([
T.StructField(“result_id”, T.StringType()),
T.StructField(“value”, T.DoubleType()),
])
)),
])
)),
])
@transform_df(
Output(“/path/to/output”),
source=Input(“/path/to/input”),
)
def compute(source):
# Apply your schema explicitly to preserve the nested Array types
return source.select(
“sample_id”,
# Transform your data to match the nested structure here
)
By defining the schema explicitly with nested ArrayType(StructType(…)) constructs, PySpark preserves the complete type hierarchy without converting nested arrays into strings. This is the standard approach for handling deeply nested complex types.
Alternative: Python UDF within Pipeline Builder
If you prefer to remain within Pipeline Builder, a Python UDF can be used to construct and return the nested structure with the desired schema. However, this approach is generally more complex than implementing the transformation in a Code Repository and is typically recommended only when there is a specific requirement to keep the logic within the visual Pipeline Builder workflow.
If you must stay within the no-code interface
There is currently no supported no-code method for creating an Array field nested inside another struct’s array. While the Add or Update Struct Field functionality supports dot notation for nested struct fields, it does not reliably preserve fields whose type is Array. Instead, those nested structures are generally serialized as strings.
For schema patterns involving nested Array types, implementing the transformation in a Python Code Repository remains the recommended and most reliable solution.
