Hey, I have a pipeline where the source data can sometimes contain sensitive data and I need to make sure that that sensitive data isn’t kept on Foundry.
I know that one option is to set up a lineage-aware retention policy to keep only the latest transaction. However, I have a pipeline downstream that I need to keep for monthly KPI tracking and it’s an incremental transform.
When the retention runs, all the downstream datasets have their previous transactions deleted, meaning that historic data is not kept in my KPI dataset.
Can I make this workflow work in any way?
I wouldn’t want to have to set up plain retention policies on every dataset because whoever creates a new dataset is bound to forget to set a retention policy to remove the sensitive data.
You can create policy overrides to accomplish what you’re describing:
In certain scenarios, you may want to stop the inheritance of deletion dates from upstream transactions and retain downstream data for longer periods. You may require this if, for example, successful aggregations or minimizations render datasets exempt from deletion requirements. For these use cases, Data Lifetime includes policy overrides. A policy override is a policy with the unique attribute of breaking the inheritance of deletion dates from upstream transactions.
Thank you, this is exactly what I was looking for.
I hadn’t been able to find it before seeing your reply, but after reading the docs you linked, it looks like the option is on the dataset where I want the chain to break. Thanks!