How can I append data from one dataset to another without overwriting the dataset that it is getting added to? If I use a union in pipeline builder, I have to overwrite the original dataset.
Hey @tillcfo, a union shouldn’t overwrite the dataset. It will add all the rows from the datasets you’re unioning together and create a separate union output of all those rows.
Docs: https://www.palantir.com/docs/foundry/pipeline-builder/transforms-union-data/
Hi Helen. That is the issue - I do not want to output a new dataset. I want the output to be the 1st dataset that rows have been appended to.
Where is the first dataset coming from in your case? Is it a transform or an upload?
The data is being ingested from an external transform.
If I’m understanding correct, in that case I don’t think you can overwrite the dataset in PIpeline Builder. Normally, you could overwrite a dataset in the outputs panel, but since you’re using that dataset also as an input, it won’t work. I would either output a new dataset with the union or see if you can union it as a part of your ingestion logic prior to creating any new pipeline.