Hi,
I’m in research to understand how other customers who has databricks and foundry use both environment together
If you are using both Foundry and Databricks then can you let me know
1- If you could successfully virtualize the data both ways
2- If you could then what is the techology and methodology you use
What we suffer is
1- Virtualize Foundry on Databricks: S3 API is not suitable for Databricks to federate via external tables on Unity Catalog. There is a way to do it on Hive but not on unity catalog
2- Virtualize Databricks on Foundry: Databricks connector doesn’t support virtualization. We utilize both S3 and ADLS Gen2 options.
Both works fine. S3 can also give unity catalog browsability but only for iceberg.
Unfortunately our version of databricks doesn’t support the iceberg
Problem with virtualizing without catalog is having to use a mapper table from unity catalog ID to the table so it is a bit cumbersome.
We use JDBC both ways which is not really ideal as thats too much data movement.
Want to get fellow databricks users opinions and experiences how they operate using 2 data platforms.