Hi Foundry community, I need your advice and help.
I’m reaching out to tap into the collective wisdom of our developer network. Over the past months, I’ve built a comprehensive machine learning project within our Foundry environment that has grown into something quite substantial. The system leverages numerous PySpark pipelines to engineer features from massive datasets, ultimately feeding a prediction module that generates insights for millions of items.
The architecture works beautifully within Foundry’s ecosystem.
However, I’m now facing an interesting challenge: we need to extract this project from Foundry to deploy it for an external customer who requires a standalone solution.
Current Architecture Overview
Our existing setup includes:
-
Multiple PySpark pipelines handling feature engineering at scale
-
Huge data transformations processing millions of records
-
Integrated ML prediction module within Foundry’s framework
-
End-to-end workflow orchestration leveraging Foundry’s native capabilities
The Migration Question
This brings me to my core question for the community: What platforms, tools, or architectural approaches would you recommend for migrating PySpark-heavy ML pipelines from Foundry with minimal refactoring effort?
Key Considerations
The ideal migration path should:
-
Preserve existing PySpark logic with minimal code changes
-
Handle millions of items efficiently (scale is crucial)
-
Support complex feature engineering workflows
-
Provide robust orchestration capabilities
-
Offer deployment flexibility for customer environments
Seeking Your Experience
I’m looking for insights from practitioners who’ve navigated similar challenges. Whether it’s cloud platforms like AWS/Azure/GCP, orchestration tools like Airflow/Prefect, or data platforms like Databricks/Snowflake—I’d love to hear about your experiences, lessons learned, and recommendations.
What platforms have worked well for you when migrating from Foundry? What unexpected challenges did you encounter? What would you do differently?
Looking forward to learning from the community’s collective experience!