Different spark profiles for incremental vs snapshot?

Ask the Community
,
1

Questions related to transforms in a spark pipeline, likely in Code Repository:
Is it possible to have different profiles for incremental builds vs snapshot builds ?

As a general usecase, having a dynamic number of executors via dynamic allocation profiles is possible, but for memory allocation there isn’t the same flexibility, hence the above questions.

To my knowledge, Spark does not currently support dynamic adjustment of memory for executors after the application has started, so it should not be possible to tweak the spark allocation after a build started.

1 Like
2

It’s has been discussed here before but it’s not possible right now. Profiles have to be fixed at CI time so you need to decide right then which profiles you want to apply, for both incremental and snapshot runs.

Here is the previous discussion

https://community.palantir.com/t/dynamic-configurations-for-incremental-pipelines/1021/2