# Dynamically switch from lightweight to spark

**URL:** <https://community.palantir.com/t/dynamically-switch-from-lightweight-to-spark/5486>\
**Category:** Ask the Community\
**Tags:** code-repositories, spark, transforms-lightweight\
**Created:** [November 20, 2025, 7:53am UTC](https://community.palantir.com/t/dynamically-switch-from-lightweight-to-spark/5486 "2025-11-20T07:53:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Soufiane](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@Soufiane](https://community.palantir.com/u/Soufiane)\
**Post date:** [November 20, 2025, 7:53am UTC](https://community.palantir.com/t/dynamically-switch-from-lightweight-to-spark/5486/1 "2025-11-20T07:53:25Z")

</div>

Hello all,

I wanted to know if there is a way to dynamically switch from a lightweight transform to a spark one ? I mean suppose we are having an incremental transform that can everyday run using lightweight, but if we want to snapshot this dataset and the size of this dataset is huge ( ~500GB ), we should do it in spark, so can we in the same code have both and depending on what we are doing switching from one to another ?

Thank you,

Soufiane

---

<div class="post-metadata">

**Author:** ![VincentF](https://avatars.discourse-cdn.com/v4/letter/v/8491ac/32.png) [@VincentF](https://community.palantir.com/u/VincentF)\
**Post date:** [November 20, 2025, 7:56am UTC](https://community.palantir.com/t/dynamically-switch-from-lightweight-to-spark/5486/2 "2025-11-20T07:56:20Z")

</div>

Not a direct answer: There is a concept of batched incremental, where only a number of transactions are processed in one given build, and this slides build after build.  
This is directly addressing this problem of “big snapshot”: https://www.palantir.com/docs/foundry/transforms-python-spark/incremental-transaction-limits

This is to my knowledge only available in spark for now, but this would be a good feature to have in lightweight too.

Second partial answer: It is possible to switch from spark to lightweight in one click in Pipeline Builder.  
I’m not aware of something equivalent in code repository.

---

<div class="post-metadata">

**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\
**Post date:** [November 15, 2026, 7:56am UTC](https://community.palantir.com/t/dynamically-switch-from-lightweight-to-spark/5486/3 "2026-11-15T07:56:42Z")

</div>

Closed on November 15, 2026, 7:56am UTC

This topic was automatically closed 360 days after the last reply. New replies are no longer allowed.
