Pipeline Builder - Locking Profiles

Ask the Community
,
1

Hi community,
as a platform architect (ADMIN), I want to have the capability to enable spark profiles in pipeline builder on demand. Users without higher level permissions like an ADMIN SHOULD NOT be able to adjust the profiles by themselves. The functionality already exists for Palantir Foundry Code Repositories.

Therefore profiles must be locked until a review from a platform team has been conducted and approved the utilization of more resources.

Does anyone has a workaround for this?
Is there a specific reason why Palantir does not provide this feature? Is this feature on the roadmap

1 Like
2

Hi Neylox,

Thanks for the suggestion! To figure out where to go from here, can you clarify a few things:

How does code repositories prevent adjusting profiles? Do you mean the import flow for “restricted profiles”?

From the Code Repos documentation:

  • Restricted Profiles: Profiles that can only be imported by administrators.

Pipeline Builder does actually support restricting the import of some compute profiles. What profile imports are you trying to restrict?

image
image1420Ă—1036 183 KB

3

Hi sperchanok,
I don’t want that end users are able to select “Extra Large” or “Large” without having special permission.

It will spinn up dynamic 2 to 128 executors.

I would wish to have similar setup like for code repository where a specific permission group is needed to enable these profiles.

As you can see in your screenshot, for the advanced part its already available, but I would consider DYNAMIC_ALLOC_MIN_2_TO_128 as something that should be restricted.

Thank you for the rapid response!

1 Like
4

Hi Neylox,

I did some more digging, and it looks like this is a known issue in our backlog. What kinds of issues are you seeing with the large and extra large profiles? If this is something you need urgently, please reach out to your Palantir administrator to contact support through internal channels. This will give us a better idea of your stack configuration and needs.

5

We have more than 10k active users on our stack and we have well established processes for spark profile enablement (code repository).

Most of the time, users don’t need more resources. We support our user base in optimizing their queries to avoid the need to increase the profiles to get more cluster resources. With this approach we can utilize our rubix cluster efficiently to deliver best in class data & ai solutions.

Without a proper review we unnecessarily block resources that might be relevant for other high impact solutions.

We filed already internally an issue and reached out the Palantir Client Support. As shown in your screenshot there are potential configuration that only need to be populated to the preset configurations. Looking forward to see this as a standard feature on our stack.

Thank you

1 Like