Hey everyone, I’m a data engineer new to the Foundry platform. I’m coming from a more traditional data stack (Airflow, dbt, Snowflake), where SQL is central to everything you do. I’ve been finding it a bit challenging to adjust to Foundry’s low-code/no-code approach, especially when it comes to building pipelines.
I’m aware that Code Repositories exist, but I wanted to ask the community (or Palantir team) whether either of the following is currently possible — and if not, whether it’s something on the roadmap.
Can I include custom SQL blocks within Pipeline Builder? It would be incredibly useful to mix custom SQL queries with low-code/no-code blocks when working with source data. I’m not talking about defining UDFs, but rather running full queries directly as part of the pipeline logic.
Is there a SQL development environment for exploration? I know about the “Explore with SQL” feature, but it seems pretty hidden. I’d love to see a more accessible, centralized interface — similar to Snowflake’s web UI — where I can run ad hoc queries at scale. Code Repositories seem focused on structured ETL jobs, not interactive exploration. My usual workflow involves querying intermediate tables and steps as I build pipelines, and I’d really benefit from an integrated query tool to support that.
Curious how others are working around this, and if there are best practices or features I might have missed.
There isn’t any way to execute custom SQL in Pipeline Builder today, adhering to the low-code / no-code philosophy. Most SQL-like data transformations should be possible with the built-in boards in Pipeline Builder; the functions documentation and AIP Assist / Generate features should be useful in determining how to achieve the desired output in such manner.
I’m not sure if what you’re referring to is the “SQL Preview” feature in Dataset Preview, but the docs for that are here. Otherwise the primary no/low-code data exploration tool that exists in Foundry is Contour, which also has a custom expression language that incorporates multiple functions from Spark SQL. Another option is Code Workspaces which gives you a more pro-code JupyterLab environment for working with datasets.
I think this question is less about low-code/no-code workflows but about SQL centric pro-code workflows.
I do agree that Foundry is lacking here and one nice/obvious solution would be to add SQL transform blocks to pipeline builder. The classical spark based sql transforms in code repository lack the iteration speed that you get in platforms like snowflake where you‘ll have an answer to your query in seconds compared to the minutes you need to wait for checks und build to complete.
We use SQL in our pipelines frequently! For standard lightweight transforms you can pick your tool flavor: DuckDB, Polars, or like, and use SQL as normal. For Spark-based transforms you can use Spark SQL.
In any case, we heavily use lightweight transforms orientated around SQL and recommend trying this approach.
@coffee-operator yes! You nailed my thoughts here perfectly. It seems like such an intuitive thing to add a SQL/Python transform node in Pipeline Builder. The work required to get a pipeline up and running in Code Repositories with SQL or Python is cumbersome, and half of my debug time is spent fixing Foundry-specific egress or package issues, when I just want to build quickly and have visibility into the outputs of what I’m building quickly. When compared to pipelining with Airflow or DBT on Snowflake or BigQuery, Foundry is lacking in my opinion
@coffee-operator This is good to know, but Code Repositories really doesn’t seem to be “quick” when trying to set something up from scratch and visualize your data/outputs along the way. Build times, errors, etc.
@achung thanks for your response and sorry to get back so late. I know these things exist, but they’re so heavyweight to get into or they’re just not robust enough for quick, iterative, SQL-based development. As a data engineer by trade, I fear Foundry will cause my tool-agnostic skills to falter
Appreciate this feedback! Our team is cooking up some ideas for improving the SQL developer experience in Foundry, particularly for more ad hoc / exploratory style workflows that need a faster entry point.
@nkaaa - would be interested to chat more about your needs. Is it ok for me to contact you directly?
I love seeing others with this feedback. Foundry is great in a lot of places but it does not shine here. The workflows are tragically slow compared to competitors.
I am optimistic this will get better with more multi modal data plane like features, but it will be painful in the meantime. More and more sql support is in the works supposedly I am crossing my fingers. Always happy to connect on topics like this from other users on the forum or palantir PD
