How to parse an array of structs in streaming pipeline builder

Ask the Community
1

I have an array of structs field and I want to flatten the column into multiple parallel arrays where each has one struct field. Is this possible in builder (streaming)? Which transform(s) would I use?

See example below.

Input Array[Struct] column:
[
{
“pose_”: {
“orientation_”: {
“w_”: “1”,
“x_”: “0”,
“y_”: “0”,
“z_”: “0”
},
“position_”: {
“x_”: “0”,
“y_”: “10”,
“z_”: “0”
}
}
},
{
“pose_”: {
“orientation_”: {
“w_”: “1”,
“x_”: “0”,
“y_”: “0”,
“z_”: “0”
},
“position_”: {
“x_”: “10”,
“y_”: “20”,
“z_”: “0”
}
}
},
{
“pose_”: {
“orientation_”: {
“w_”: “1”,
“x_”: “0”,
“y_”: “0”,
“z_”: “0”
},
“position_”: {
“x_”: “10”,
“y_”: “50”,
“z_”: “0”
}
}
}
]

Output Array[double] columns:
pose_orientation_w: [1,1,1]
pose_orientation_x: [0,0,0]
pose_orientation_y: [0,0,0]
pose_orientation_z: [0,0,0]
pose_position_x: [0,10,10]
pose_position_y: [10,20,50]
pose_position_z: [0,0,0]

2

Hi @groth you can use a combination of the “Transform array element” and “Get struct field” expressions:

Notional example:

image
image1015×411 31.8 KB

You will need to repeat this board several times to extract all of the values you want.

2 Likes
3

Hey @dherls! Let’s assume I did not know how many elements would be in the resulting after the first step but they would all have the same schema for the key: value pairs. Is there any way to grab all of those pairs without manually locating them?

4

Hi @asohrabi pipeline builder unfortunately does not allow for dynamic column creation, so you would need to know all of the locators in advance