Adding easy SQL querying in foundry

Product Feedback
,
1

Feature Request: Native Querying in Pipeline Builder & Ontology

Summary
Enable users to run SQL queries directly within the Pipeline Builder and Ontology interfaces to support debugging, QA, and data exploration without relying on external data source previews.

Problem Statement

Users currently lack the ability to easily inspect or query data at intermediate steps in a pipeline or directly within the ontology. As a result:

  • Debugging pipeline issues becomes time-consuming.

  • Users must repeatedly switch to underlying data sources to run checks.

  • The existing SQL Preview is unreliable for complex queries, joins, and deeper data validation.

  • Overall workflow efficiency suffers due to limited visibility into data transformations.

Requested Enhancements

1. Querying in Pipeline Builder

Provide the ability to:

  • Run SQL queries directly on outputs of individual steps/nodes.

  • Inspect intermediate data for QA and debugging.

  • Validate logic without executing the entire pipeline.

  • Use simple SQL syntax for fast ad-hoc checks.

2. Querying in Ontology

Enable:

  • Direct SQL querying on ontology entities and relationships.

  • More robust, accurate alternative to the current SQL Preview.

  • Support for complex queries, joins, and data validation within the ontology interface.

  • Consistent visibility into how ontology structures map to actual data.

Value / Impact

  • Dramatically reduces debugging time for pipeline development.

  • Improves reliability of QA workflows.

  • Eliminates friction caused by switching to backend data sources.

  • Empowers users with deeper insight into their data models and pipeline logic.

  • Supports more advanced analyses without requiring external tools.

1 Like
2

Hi @rsfogel ,

I can answer for the Ontology side.

Your ask totally makes sense. In fact, we are currently developing a SQL Preview for ontology objects, where the same way than with datasets, you can query your ontology objects with SQL. This capability to natively query the ontology storage with sql is already released today within AIP Analyst, AIP Agents and AIP Logic with the “Query with SQL” tool. This is very much in development, we are focusing on improving performance, stability, and our Spark SQL dialect coverage.

It will definitely eliminate the friction of switching datasources and apps since the places to author SQL will not be datasource-dependent. Both ontology objects and datasets can be queried.

I hope this answers your question.

3 Likes