Lightweight transform taking longer than expected to build

Ask the Community
1

Hello, I am currently working on implementing a mixed integer programing problem to find optimal actions in a dataset. Due to the format of the program I believe it can not be run in the modeling objectives. Currently I have the function running in a lightweight transform, however the build time is very slow (build sometimes takes 30 minutes sometimes 5 hours, and on a local machine takes about 10). Additionally the input dataset is roughly 500k rows and pandas is being used within the implementation.

I am wondering if there is any reason that the build time might be so long and if it is a problem related to light transforms or if this problem would likely be solved by transitioning the pandas implementation to a polars implementation or similar.

2

Is the resource configuration the same from your local machine to the lightweight transforms? Eg are the memory and core configuration the same/similar. You can do this with @lightweight(cpu_cores=8, memory_gb=16) (docs).

3

Hi Ben thanks for the response,
The configuration of the machines is not the same but the lightweight transform has more resources than the local machine (128gb 2 cores for the lightweight, 32gb 1 core for the local machine). I have also tried reducing the cores and memory for the lightweight transform to match the local machine as well but that did not work.