[experiment/test] SQLite Bucket Storage (and MySQL bucket storage)

[stevensJourney]

Background

This PR is just an experiment at this stage. The goal is to share the results of a test. What better place to start a discussion that a PR with a working implementation :D

Overview

When self hosting or using our Docker images for local development: devs currently have the option of using MongoDB or Postgres as the Sync bucket storage database. These are our most common data source connections which means devs are quite likely to have access to a DB to use for sync storage. For other connections, like Convex, MSSQL or MySQL that might not be the case. Having to configure a MongoDB replica set or Postgres database can be a pain.

For local development, the requirement of starting a sync bucket storage database usually complicates the setup quite a bit. Typically a Docker compose project is the solution to this, but, it could be nice if one was able to use just the PowerSync service Docker image (without Docker compose) and the dev's supplied source database.

This PR tries to simplify those flows by adding SQLite as an option for bucket storage. The option of using a memory or file-based database are provided.

The experiment part of this PR is that we add SQLite support by first attempting to create a sync bucket storage implementation which uses a somewhat generic ORM interface. This is essentially a port of the current Postgres bucket storage implementation, but ported to use MikroORM for the database in the various implementation classes... Under the hood we start out by giving MikroORM a SQLite driver which delegates the typed mutations to SQLite. This PR tries to share some implementation while allowing for escaping out of MikroORM to raw SQL in some areas - e.g the Compacting is currently mainly implemented in raw SQL.

Why share some implementation? That's also part of this experiment. The goal is to see what it could look like to have a shared sync bucket storage implementation which could be compatible with multiple SQL drivers. As a test, this PR actually adds both SQLite and MySQL sync bucket storage implementations - where minimal MySQL driver code is added (for now at least). It's still an open question if this is maintainable, scalable or performant enough.

Why MikroORM? There is no real strong reason to choose MikroORM at this point. This could theoretically also be implemented in Drizzle - it would actually be nice to see how the two would compare at this. It seems like MikroORM has a somewhat nice method of generically defining schemas, while Drizzle seems to be more explicit about defining a schema for each possible driver (that might not be a bad thing).

Status

For SQLite: all our common/shared sync bucket storage tests currently pass. It's usually also a good test for a sync bucket storage implementation to be tested as the storage during a replicator module's integration tests. The SQLite storage has been tested in the Postgres replication tests - with all tests passing there also.

MySQL support has only been added recently. I've only verified that the storage integration tests pass and it works in a demo application.

The implementations here are not optimised much. No benchmarks have been ran yet. It does seem like the SQLite tests complete quite quickly though. For SQLite, there are some known performance issues: mainly that MikroORM does not yet use Workers to delegate multiple connections - which essentially prevents concurrency.

Other caveats:

SQLite runs either in memory or with a local file database. This means it can only be used in a single instance (not with multiple pods/runners) started in the unified (both API and replication) mode.

Usage

The module-mikroorm-storage README file contains some detailed notes. Generally, this is currently exposed as a new option in the self hosted PowerSync config file

replication:
  connections:
    - type: postgresql
      uri: postgres://postgres:mypassword@localhost:5432/postgres
      sslmode: disable

storage:
  type: mikroorm:sqlite
  filename: ':memory:' 

---

AI Usage disclaimer: The code generated here was made by first creating a detailed plan with Codex GPT 5.5. The plan is actually committed in the plans/ folder. Through many conversations and feedback rounds, we eventually got to this point.

[changeset-bot]

⚠️ No Changeset found

Latest commit: e71d390

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR