Auto register tables from 1 dataset in BigQuery project

Ask the Community
,
1

Hi,

I am trying to use virtual tables to query tables from 1 dataset in our BigQuery project. Our BQ project contains a lot of datasets and right now with auto registration, it tries to register all tables in all datasets and times out.

Is there a way to specify the dataset when auto registering virtual tables? I know it can be done manually per table but every day a table is added to the dataset so I would like this to be done automatically.

Thanks for any help!

2

Are you able to scope down the credentials on your source (or create a second source with scoped-down credentials) just to the datasets you want to auto-register?

Looking ahead, we are also exploring creating a public API for virtual table registration that would provide more granular control for managing table registration. E.g. if you wanted to set up a cron job to register tables into Foundry on a schedule. You could then write logic against BigQuery to identify the tables to register as part of that job.

1 Like
3

Creating a second source with scoped down credentials worked, thanks!

1 Like