Skip to content

Adjust to provide for a generic database engine type #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
realtyem wants to merge 4 commits into
base: psycopg3
Choose a base branch
from
Draft

Adjust to provide for a generic database engine type #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7310394
Adjust to provide for a generic database engine type
realtyem Nov 26, 2025
90af5cb
Attempt at docs adjustment
realtyem Nov 27, 2025
274e7cc
Fix up for Complement. Caught an issue where worker mode would have o…
realtyem Nov 27, 2025
009b4d3
Experiment to try and make the module load
realtyem Dec 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -684,16 +684,16 @@ jobs:
matrix:
include:
- arrangement: monolith
database: SQLite
database: sqlite

- arrangement: monolith
database: Postgres
database: postgres

- arrangement: workers
database: Postgres
database: postgres

- arrangement: workers
database: Psycopg
database: psycopg

steps:
- name: Checkout synapse codebase
Expand Down Expand Up @@ -721,7 +721,7 @@ jobs:
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | synapse/.ci/scripts/gotestfmt
shell: bash
env:
POSTGRES: ${{ (matrix.database == 'Postgres' || matrix.database == 'Psycopg') && matrix.database || '' }}
POSTGRES: ${{ (matrix.database == 'postgres' || matrix.database == 'psycopg') && matrix.database || '' }}
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
name: Run Complement Tests

Expand Down
2 changes: 1 addition & 1 deletion docker/complement/conf/start_for_complement.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export SYNAPSE_REPORT_STATS=no


case "$SYNAPSE_COMPLEMENT_DATABASE" in
postgres)
psycopg2)
# Set postgres authentication details which will be placed in the homeserver config file
export POSTGRES_DRIVER=psycopg2
export POSTGRES_PASSWORD=somesecret
Expand Down
3 changes: 2 additions & 1 deletion docker/conf/homeserver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ listeners:

{% if POSTGRES_PASSWORD %}
database:
name: "{{ POSTGRES_DRIVER or "psycopg2" }}"
name: "postgres"
driver: "{{ POSTGRES_DRIVER or "psycopg2" }}"
args:
user: "{{ POSTGRES_USER or "synapse" }}"
password: "{{ POSTGRES_PASSWORD }}"
Expand Down
4 changes: 2 additions & 2 deletions docs/other/running_synapse_on_single_board_computers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ limit_remote_rooms:
# Database configuration
database:
# Use postgres for the best performance
name: psycopg2
name: postgres
args:
user: matrix-synapse
# Generate a long, secure password using a password manager
Expand All @@ -72,4 +72,4 @@ admin@homeserver:~$ sudo --user postgres psql matrix-synapse --command 'select c
#irc:matrix.org | 461 | 3751
#decentralizedweb-general:matrix.org | 997 | 1509
#whatsapp:maunium.net | 554 | 854
```
```
4 changes: 2 additions & 2 deletions docs/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ section in your config file to match the following lines:

```yaml
database:
name: psycopg2
name: postgres
args:
user: <user>
password: <pass>
Expand All @@ -72,7 +72,7 @@ database:
cp_max: 10
```

All key, values in `args` are passed to the `psycopg2.connect(..)`
All key, values in `args` are passed to the appropriate modules `connect()`
function, except keys beginning with `cp_`, which are consumed by the
twisted adbapi connection pool. See the [libpq
documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
Expand Down
4 changes: 2 additions & 2 deletions docs/usage/administration/monitoring/reporting_homeserver_usage_statistics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The following statistics are sent to the configured reporting endpoint:
| `python_version` | string | The Python version number in use (e.g "3.7.1"). Taken from `sys.version_info`. |
| `total_users` | int | The number of registered users on the homeserver. |
| `total_nonbridged_users` | int | The number of users, excluding those created by an Application Service. |
| `daily_user_type_native` | int | The number of native, non-guest users created in the last 24 hours. |
| `daily_user_type_native` | int | The number of native, non-guest users created in the last 24 hours. |

@realtyem realtyem Nov 27, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not purposely change anything here, I suspect some kind of automated whitespace adjustment

| `daily_user_type_guest` | int | The number of guest users created in the last 24 hours. |
| `daily_user_type_bridged` | int | The number of users created by Application Services in the last 24 hours. |
| `total_room_count` | int | The total number of rooms present on the homeserver. |
Expand All @@ -49,7 +49,7 @@ The following statistics are sent to the configured reporting endpoint:
| `r30v2_users_web` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "mozilla" or "gecko" (case-insensitive) in the user agent string. |
| `cache_factor` | int | The configured [`global factor`](../../configuration/config_documentation.md#caching) value for caching. |
| `event_cache_size` | int | The configured [`event_cache_size`](../../configuration/config_documentation.md#caching) value for caching. |
| `database_engine` | string | The database engine that is in use. Either "psycopg2" meaning PostgreSQL is in use, or "sqlite3" for SQLite3. |
| `database_engine` | string | The database engine that is in use. Either "psycopg2" or "psycopg" meaning PostgreSQL is in use, or "sqlite3" for SQLite3. |
| `database_server_version` | string | The version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system. |
| `log_level` | string | The log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc. |

Expand Down
24 changes: 20 additions & 4 deletions docs/usage/configuration/config_documentation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1466,7 +1466,9 @@ For more information on using Synapse with Postgres, see [here](../../postgres.m

This setting has the following sub-options:

* `name` (string): This option specifies the database engine to use: either `sqlite3` (for SQLite) or `psycopg2` (for PostgreSQL). `psycopg` references the experimental psycopg3 driver, which may be used as a drop in replacement for `psycopg2`. If no name is specified Synapse will default to SQLite. Defaults to `"sqlite3"`.
* `name` (string): This option specifies the database engine to use: either `sqlite3` (for SQLite) or `postgres` (for PostgreSQL). `psycopg2` can be used for backwards compatibility for a specific driver. See the 'driver' section for more information. If no name is specified Synapse will default to SQLite. Defaults to `"sqlite3"`.

* `driver` (string): The specific database library "driver" to use. The default for the SQLite based system is `sqlite3`. The default for the PostgreSQL based system is `psycopg2`. `psycopg` references the experimental psycopg3 driver, which may be used as a drop in replacement for `psycopg2`. This setting is not required as the `name` setting guides its default. Only set and change this if you know what you are doing!

* `txn_limit` (integer): Gives the maximum number of transactions to run per connection before reconnecting. 0 means no limit. Defaults to `0`.

Expand All @@ -1487,7 +1489,7 @@ database:

```yaml
database:
name: psycopg2
name: postgres
txn_limit: 10000
args:
user: synapse_user
Expand All @@ -1498,6 +1500,20 @@ database:
cp_min: 5
cp_max: 10
```

```yaml
database:
name: postgres
driver: psycopg
args:
user: synapse_user
password: secretpassword
dbname: synapse
host: localhost
port: 5432
cp_min: 5
cp_max: 10
```
---
### `databases`

Expand Down Expand Up @@ -1550,7 +1566,7 @@ Example configuration:
```yaml
databases:
basement_box:
name: psycopg2
name: postgres
txn_limit: 10000
data_stores:
- main
Expand All @@ -1563,7 +1579,7 @@ databases:
cp_min: 5
cp_max: 10
my_other_database:
name: psycopg2
name: postgres
txn_limit: 10000
data_stores:
- state
Expand Down
37 changes: 25 additions & 12 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ postgres = [
"psycopg2cffi>=2.8;platform_python_implementation == 'PyPy'",
"psycopg2cffi-compat==1.1;platform_python_implementation == 'PyPy'",
]
psycopg = ["psycopg>=3.1"]
psycopg-dev = ["psycopg[c] (>=3.1)"]
saml2 = ["pysaml2>=4.5.0"]
oidc = ["authlib>=0.15.1"]
# systemd-python is necessary for logging to the systemd journal via
Expand Down Expand Up @@ -140,7 +140,7 @@ all = [
"psycopg2cffi>=2.8;platform_python_implementation == 'PyPy'",
"psycopg2cffi-compat==1.1;platform_python_implementation == 'PyPy'",
# experimental psycopg for postgres
"psycopg>=3.1",
"psycopg[c] (>=3.1)",
# saml2
"pysaml2>=4.5.0",
# oidc and jwt
Expand Down
37 changes: 30 additions & 7 deletions schema/synapse-config.schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1818,7 +1818,7 @@ properties:
- name: sqlite3
args:
database: /path/to/homeserver.db
- name: psycopg2
- name: postgres
txn_limit: 10000
args:
user: synapse_user
Expand All @@ -1828,6 +1828,16 @@ properties:
port: 5432
cp_min: 5
cp_max: 10
- name: postgres
driver: psycopg
args:
user: synapse_user
password: secretpassword
dbname: synapse
host: localhost
port: 5432
cp_min: 5
cp_max: 10
databases:
type: object
description: >-
Expand Down Expand Up @@ -1922,7 +1932,7 @@ properties:
default: {}
examples:
- basement_box:
name: psycopg2
name: postgres
txn_limit: 10000
data_stores:
- main
Expand All @@ -1935,7 +1945,7 @@ properties:
cp_min: 5
cp_max: 10
my_other_database:
name: psycopg2
name: postgres
txn_limit: 10000
data_stores:
- state
Expand Down Expand Up @@ -5880,13 +5890,26 @@ $defs:
enum:
- sqlite3
- psycopg2
- psycopg
- postgres
description: >-
This option specifies the database engine to use: either `sqlite3`
(for SQLite) or `psycopg2` (for PostgreSQL). `psycopg` references the
experimental psycopg3 driver, which may be used as a drop in replacement
for `psycopg2`. If no name is specified Synapse will default to SQLite.
(for SQLite) or `postgres` (for PostgreSQL). `psycopg2` can be used for
backwards compatibility for a specific driver. See the 'driver' section for
more information. If no name is specified Synapse will default to SQLite.
default: sqlite3
driver:
type: string
enum:
- sqlite3
- psycopg2
- psycopg
description: >-
The specific database library "driver" to use. The default for the SQLite
based system is `sqlite3`. The default for the PostgreSQL based system is
`psycopg2`. `psycopg` references the experimental psycopg3 driver, which may
be used as a drop in replacement for `psycopg2`. This setting is not required
as the `name` setting guides its default. Only set and change this if you
know what you are doing!
txn_limit:
type: integer
description: >-
Expand Down
11 changes: 8 additions & 3 deletions scripts-dev/complement.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,8 +270,13 @@ main() {
# Pass through the workers defined. If none, it will be an empty string
export PASS_SYNAPSE_WORKER_TYPES="$WORKER_TYPES"

# Workers can only use Postgres as a database.
export PASS_SYNAPSE_COMPLEMENT_DATABASE=postgres
# Workers can only use Postgres as a database. If a specific driver is
# not requested, fallback to psycopg2 as a default
if [[ "$POSTGRES" = "psycopg" ]]; then
export PASS_SYNAPSE_COMPLEMENT_DATABASE=psycopg
else
export PASS_SYNAPSE_COMPLEMENT_DATABASE=psycopg2
fi

# And provide some more configuration to complement.

Expand All @@ -284,7 +289,7 @@ main() {
if [[ "$POSTGRES" = "psycopg" ]]; then
export PASS_SYNAPSE_COMPLEMENT_DATABASE=psycopg
elif [[ -n "$POSTGRES" ]]; then
export PASS_SYNAPSE_COMPLEMENT_DATABASE=postgres
export PASS_SYNAPSE_COMPLEMENT_DATABASE=psycopg2
else
export PASS_SYNAPSE_COMPLEMENT_DATABASE=sqlite
fi
Expand Down
12 changes: 11 additions & 1 deletion synapse/config/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,24 @@ class DatabaseConnectionConfig:
def __init__(self, name: str, db_config: dict):
db_engine = db_config.get("name", "sqlite3")

if db_engine not in ("sqlite3", "psycopg2", "psycopg"):
if db_engine not in ("sqlite3", "psycopg2", "postgres"):
raise ConfigError("Unsupported database type %r" % (db_engine,))

if db_engine == "sqlite3":
db_config["driver"] = "sqlite3"

@realtyem realtyem Nov 27, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is guarding against trying to use, for example, aiosqlite or something. No spoilers!

db_config.setdefault("args", {}).update(
{"cp_min": 1, "cp_max": 1, "check_same_thread": False}
)

# Allow for a generic "postgres". Then, can specify a driver directly to
# override. For backwards compatibility, allow this to remain "psycopg2" for
# now. Setting the driver below is the important detail.
elif db_engine in ("postgres", "psycopg2"):
# Default to "psycopg2"
db_driver = db_config.setdefault("driver", "psycopg2")
if db_driver not in ("psycopg2", "psycopg"):
raise ConfigError("Unsupported database driver %r" % (db_driver,))

data_stores = db_config.get("data_stores")
if data_stores is None:
data_stores = ["main", "state"]
Expand Down
2 changes: 1 addition & 1 deletion synapse/storage/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ def _on_new_connection(conn: Connection) -> None:
)

connection_pool = adbapi.ConnectionPool(
db_config.config["name"],
db_config.config["driver"],
cp_reactor=reactor,
cp_openfun=_on_new_connection,
**db_args,
Expand Down
Loading
Loading