Hello Everyone ! I am doing my first speedrun for E2E pipeline on AIP. Facing issues with creating a Master DataSet using union. I have those columns as required in the dataset and hence it should not be giving me a “Missing Required Column” error. As you can see in the picture(1) it displays the error. In Picture(2) and Picture (3)it displays the columns in datasets are present for both. I do not know what I may be doing wrong here, but just wanted to ask for any guidance if anyone knows anything about this. Thanks !
Hey @VedantGosavi - It looks like the schemas of the two inputs to the union are slightly different (A union requires that all inputs have the same schema. If input schemas do not all match, the union will display an error message with a list of missing columns).
Specifically, the Join Office Goods input has a column called due_date_time that is not present in the Join Bureau input, and the Join Bureau input has a column order_due_date that is not in the Join Office Goods input. To resolve this, you can either drop both columns or rename one of them so that they have matching names (due_date_time or order_due_date).
1 Like