VELOX vs GLUTEN

[ZygD]

So, in Foundry, we can explicitly use BOTH references (GLUTEN as profile and VELOX as backend):

@configure(
   ["EXECUTOR_MEMORY_MEDIUM", "EXECUTOR_MEMORY_OFFHEAP_FRACTION_HIGH", "GLUTEN"],
   backend=ComputeBackend.VELOX)
@transform_df(
   Output('/Project/folder/output'),
   source_df=Input('/Project/folder/input'),
)
def compute(source_df):
   ...

This is strange, because…

• GLUTEN profile itself has a setting enabling VELOX backend:
  

  

• The official documentation does not use the GLUTEN profile only referencing the VELOX backend. It implies that Gluten is used under the hood.

I’m lost here. What should we use: both, one of them? which? What if we use both, some particular setting comes from both and differs?

[tedcj]

Hey @ZygD ,

Thanks for raising this, it is indeed confusing! You can disregard the GLUTEN profile entirely, it is functionally equivalent to selecting the backend as VELOX and should be deprecated to avoid such confusion.