I’m using incremental transforms in Foundry to build a historical dataset based on weekly .xlsx snapshot files ingested via an SFTP connector. The incremental logic works well for appending new data and maintaining history.
However, I now need to remove several columns from the current dataset schema.
Is there a recommended way to drop columns from an incremental dataset without losing any previously stored data (other than the removed columns)?
I need to preserve all historical records while performing this schema change safely.
Any best practices or example approaches would be greatly appreciated.