# Deleting Files in historic transactions without breaking the incremental

**URL:** <https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824>\
**Category:** Ask the Community\
**Tags:** code-repositories, transforms-python\
**Created:** [July 24, 2024, 9:46am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824 "2024-07-24T09:46:04Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicornk](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/nicornk/32/192_2.png) [@nicornk](https://community.palantir.com/u/nicornk)\
**Post date:** [July 24, 2024, 9:46am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/1 "2024-07-24T09:46:04Z")

</div>

I have an incremental transform writing out daily partitioned parquet files based on an incremental ingestion. (\>10TB)

Due to Business Requirements there might be corrections for past dates.

Therefore, I would like to - within my incremental transform - delete special parquet files and add others replacing the data.

To my understanding this common requirement is not supported and there is also no workaround possible that keeps the incremental behaviour intact.

Did anyone solve this challenge?

---

<div class="post-metadata">

**Author:** ![md5](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/md5/32/667_2.png) [@md5](https://community.palantir.com/u/md5)\
**Post date:** [July 24, 2024, 10:43am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/2 "2024-07-24T10:43:39Z")

</div>

I’ve got a response in the past from Palantir that it’s not possible, but actually there is a workaround that we are utilising in one of our pipelines where we need to re-compute some already processed historical data, and the easiest is to remove the corrupted files from the view.

Let’s say your dataset is` ri.foundry.main.dataset.c4385d1d-67d0-46f5-984f-b391sdada3` and you want to remove a list of files or a full partition  
so that the incrementallity is preserved, then you need to:

1. Have a process that performs the delete

```python
    delete_files(
        dataset_rid="ri.foundry.main.dataset.c4385d1d-67d0-46f5-984f-b391sdada3",
        branch="master",
        remove_files=[
            "spark/year=2024/month=4/day=29/hour=15/part-00000-b33bb585-e338-4c31-adbb-431bed04bbb5.c000.snappy.parquet"
        ]

```

where the delete\_files does:

- Remove files from a datasets view with a DELETE transaction:

```python
  def remove_files(
        self,
        dataset: "Dataset",
        logical_paths: List[str]
):
        transaction = self.start_transaction(
            dataset=dataset,
            txn_type=TransactionType.DELETE
        )
        self._catalog_service.add_files_to_delete_transaction(
            auth_header=self.ctx.auth_token,
            dataset_rid=str(dataset.rid),
            transaction_rid=str(transaction.rid),
            request=LogicalFilesRequest(
                logical_paths=logical_paths
            )
        )
        self.commit_transaction(txn=transaction)

```

- Forces a new build on your dataset

```python
client._build_manager_service.submit_build(
auth_header=token,
submit_build_request=create_submit_build_request(_dataset.rid)
)

```

the force build is needed as the dataset where you delete files from needs to have an empty append transaction to preserve the incrementality

so in your transformer you would need to add logic that checks if the last transaction was delete and just do an empty append transaction if so (you end-up with two transactions! DELETE + APPEND), with next scheduled run you’ll end up in the else condition and the job will work as usual

```python
    if is_last_transaction_delete(
        input_dataset=input_dataset, token=ctx.auth_header
    ):
        output.set_mode("append")
    else:
        # all my regular spark logic

```

hope this helps, and that you’ll be able to fill the missing calls and decrypt the provided snippets 😉

ps: @palantir make this somehow easier for us 🙏 a simple parameter allow\_delete for `@incremental` would be the best

---

<div class="post-metadata">

**Author:** ![nicornk](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/nicornk/32/192_2.png) [@nicornk](https://community.palantir.com/u/nicornk)\
**Post date:** [July 25, 2024, 11:33am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/3 "2024-07-25T11:33:59Z")

</div>

Thanks for providing this solution. It seems like the incremental of the “large” dataset is still working with your workaround, however incremental pipelines build on top of that fail ☹

So I agree that Palantir should offer an allow\_delete parameter to make this workflow possible.

 ![image](https://canada1.discourse-cdn.com/palantir/original/1X/02036443a1deaa2117f3b581b5782d9b0bad182d.jpeg)

---

<div class="post-metadata">

**Author:** ![priyanshukm](https://avatars.discourse-cdn.com/v4/letter/p/c0e974/32.png) [@priyanshukm](https://community.palantir.com/u/priyanshukm)\
**Post date:** [July 30, 2024, 9:43am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/4 "2024-07-30T09:43:00Z")

</div>

A flag called `allow_retention` exists in python transforms and `IGNORE_INCREMENTAL_DELETES` in java transform that if added to transform does not break incrementality if DELETE transaction is added to the dataset.

---

<div class="post-metadata">

**Author:** ![nicornk](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/nicornk/32/192_2.png) [@nicornk](https://community.palantir.com/u/nicornk)\
**Post date:** [July 30, 2024, 12:44pm UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/5 "2024-07-30T12:44:13Z")

</div>

Does this work for all DELETES or only for deletes by the retention service?

---

<div class="post-metadata">

**Author:** ![priyanshukm](https://avatars.discourse-cdn.com/v4/letter/p/c0e974/32.png) [@priyanshukm](https://community.palantir.com/u/priyanshukm)\
**Post date:** [July 31, 2024, 12:01pm UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/6 "2024-07-31T12:01:25Z")

</div>

It should work for any DELETE as long as you write metadata to the deleted transaction that it was deleted via the retention service.

```auto
{
  "deletion-metadata": {
    "reason": "EXPIRED"
  }
}

```

It’s easier if you want to delete the entire transaction (i.e. all files in it). If you use retention service to mark the transaction, it will perform all the steps you described above (creating txn/adding files/committing txn) and writing appropriate transaction metadata to allow `allow_retention` and `IGNORE_INCREMENTAL_DELETES` to work as expected.

---

<div class="post-metadata">

**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\
**Post date:** [January 8, 2025, 5:14pm UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/7 "2025-01-08T17:14:20Z")

</div>

Automatically bumped on January 8, 2025, 5:14pm UTC



---

<div class="post-metadata">

**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\
**Post date:** [June 26, 2025, 3:47am UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/8 "2025-06-26T03:47:03Z")

</div>

Automatically bumped on June 26, 2025, 3:47am UTC



---

<div class="post-metadata">

**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\
**Post date:** [August 26, 2026, 12:41pm UTC](https://community.palantir.com/t/deleting-files-in-historic-transactions-without-breaking-the-incremental/824/9 "2026-08-26T12:41:49Z")

</div>

Automatically bumped on August 26, 2026, 12:41pm UTC


