# Different spark profiles for incremental vs snapshot?

**URL:** <https://community.palantir.com/t/different-spark-profiles-for-incremental-vs-snapshot/3875>\
**Category:** Ask the Community\
**Tags:** code-repositories, transforms-python-builds\
**Created:** [May 15, 2025, 11:58am UTC](https://community.palantir.com/t/different-spark-profiles-for-incremental-vs-snapshot/3875 "2025-05-15T11:58:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![VincentF](https://avatars.discourse-cdn.com/v4/letter/v/8491ac/32.png) [@VincentF](https://community.palantir.com/u/VincentF)\
**Post date:** [May 15, 2025, 11:58am UTC](https://community.palantir.com/t/different-spark-profiles-for-incremental-vs-snapshot/3875/1 "2025-05-15T11:58:00Z")

</div>

Questions related to transforms in a spark pipeline, likely in Code Repository:  
**Is it possible to have different profiles for incremental builds vs snapshot builds ?**

As a general usecase, having a dynamic number of executors via dynamic allocation profiles is possible, but for memory allocation there isn’t the same flexibility, hence the above questions.

To my knowledge, Spark does not currently support dynamic adjustment of memory for executors after the application has started, so it should not be possible to tweak the spark allocation after a build started.

---

<div class="post-metadata">

**Author:** ![nicornk](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/nicornk/32/192_2.png) [@nicornk](https://community.palantir.com/u/nicornk)\
**Post date:** [May 15, 2025, 5:40pm UTC](https://community.palantir.com/t/different-spark-profiles-for-incremental-vs-snapshot/3875/2 "2025-05-15T17:40:50Z")

</div>

It’s has been discussed here before but it’s not possible right now. Profiles have to be fixed at CI time so you need to decide right then which profiles you want to apply, for both incremental and snapshot runs.

Here is the previous discussion

https://community.palantir.com/t/dynamic-configurations-for-incremental-pipelines/1021/2

---

<div class="post-metadata">

**Author:** ![system](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/system/32/3_2.png) [@system](https://community.palantir.com/u/system)\
**Post date:** [July 14, 2025, 5:41pm UTC](https://community.palantir.com/t/different-spark-profiles-for-incremental-vs-snapshot/3875/3 "2025-07-14T17:41:02Z")

</div>

Closed on July 14, 2025, 5:41pm UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.