Skip to content

[BUG] A lot of transactions being retried #3002

Description

@wbyoung

Describe the bug

In development, when I quickly add a lot of tuples to rebuild the state of my system from the ground up, I'm getting a lot of warnings:

permify-565bc8487f-fpdnx permify time=2026-06-11T19:52:55.466Z level=WARN msg="serialization error occurred" tenant_id=t1 retry=2
permify-565bc8487f-fpdnx permify time=2026-06-11T19:52:55.467Z level=WARN msg="waiting before retry" tenant_id=t1 backoff_duration=110

And from PostgreSQL:

db-0 db 2026-06-11 19:52:55.248 UTC [38833] STATEMENT:  UPDATE relation_tuples SET expired_tx_id = $1 WHERE expired_tx_id = $2 AND tenant_id = $3 AND entity_id = $4 AND entity_type = $5 AND relation = $6 AND subject_type = $7
db-0 db 2026-06-11 19:52:55.250 UTC [38844] ERROR:  could not serialize access due to read/write dependencies among transactions
db-0 db 2026-06-11 19:52:55.250 UTC [38844] DETAIL:  Reason code: Canceled on commit attempt with conflict in from prepared pivot.
db-0 db 2026-06-11 19:52:55.250 UTC [38844] HINT:  The transaction might succeed if retried.
db-0 db 2026-06-11 19:52:55.250 UTC [38844] STATEMENT:  commit

To Reproduce
Steps to reproduce the behavior:

  • Write a lot of tuples quickly. I may be able to dig in and try to create a more concise example if required, but thought I might get a little feedback first.

Expected behavior
I understand this may be working as expected since they're being retried, but it seems to be happening quite a bit more than I would expect. (Maybe I just need to learn more about how things work.)

Additional context
Add any other context about the problem here.

Environment (please complete the following information, because it helps us investigate better):

  • OS: Fedora 43
  • Version: v1.7.0, specifically ghcr.io/permify/permify@sha256:e56df1a36c76ddffdf4467dbe2078daa35a54322f405217e29c0f8c8a4bfe9ff
  • Docker version 29.5.2, build 79eb04c
  • Kubernetes Client Version: v1.35.4
  • Kubernetes Server Version: v1.34.1

Metadata

Metadata

Labels

bugSometing isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions