@tedcj @matthewbayer Awesome talk at DevCon 4. Would it be possible to share the general logic used for the reference example of pulling data from S3 into Foundry Iceberg tables with external transforms?
I’m particularly interested in the logic used for row-level edits (sounds like potentially a one-step process) and the Iceberg storage option used. We have large tables involving row-level deletes and edits; it’s tricky to handle incremental pipelines for these. We’ve been waiting for the general Iceberg roll-out and are extremely interested in having a setup similar to what you’ve described for our stack - it would be awesome to leverage your approach as the starting point.
https://www.youtube.com/watch?v=7QzrPRM8Bqw
https://www.palantir.com/docs/foundry/iceberg/managed-virtual