Sqlite as a data source?

Ask the Community
1

I could not find any resource on what would be the best way to use sqlite as a data source (either batch or live sync).
Any idea/recommendation except for moving sqlite data to a supported database (e.g., PostgreSQL)?
Thx!

2

There are two options.

  1. Install an Agent on the (virtual) machine where the SQLite file is located (or accessible via a mounted network drive) and connect via an Agent worker connection.
  2. Ingest the SQLite file into a schemaless dataset in Foundry with a file-based sync using a source such as Directory, FTP, SFTP, or SMB and then use unstructured file transforms downstream to extract the tables from the file, convert them to Spark dataframes, and write them to the transform outputs.

Both of these options should work well, but the first allows you to do everything without writing any code, so it is probably the simplest!

1 Like
3

Another option would be to use https://litestream.io/ to stream changes from your sqlite file to a schemaless dataset by using the s3 compatible API.

We have some of that integrations in production running fine for some time.

3 Likes