Skip to content

feat(migrations): expose queryable resource links#12193

Merged
abnegate merged 15 commits into
mainfrom
feat/migration-resource-attrs
Jul 20, 2026
Merged

feat(migrations): expose queryable resource links#12193
abnegate merged 15 commits into
mainfrom
feat/migration-resource-attrs

Conversation

@abnegate

@abnegate abnegate commented May 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Replaces compound/root migration selectors with the canonical Appwrite identity shape: resourceId, resourceInternalId, resourceType, parentResourceId, parentResourceInternalId, and parentResourceType.
  • Adds destinationResourceId, destinationResourceInternalId, and destinationResourceType so a restored or imported database can be linked to its exact generation.
  • Makes all public and internal relation fields queryable and indexed.
  • Passes all six source-selector fields by name to utopia-php/migration 1.18.0.
  • Records exact source, parent, and destination sequences at CSV/JSON migration creation.
  • Uses one bulk schema call in V25 and keeps legacy composite values unresolved unless both resource generations can be safely identified.
  • Rejects historical internal-ID backfills when the current resource was created after the migration, preventing public-ID reuse from linking an old migration to a replacement.
  • Treats both null and empty optional relation fields as missing during idempotent backfill.
  • Runs source/destination success hooks and export finalization before persisting completed.

Database lookup contract

For an exact current database generation, query an OR across these relation triples:

  • resourceType + resourceId + resourceInternalId
  • parentResourceType + parentResourceId + parentResourceInternalId
  • destinationResourceType + destinationResourceId + destinationResourceInternalId

A public-ID-only query intentionally returns history across generations. TablesDB/legacy roots use database; DocumentsDB and VectorsDB use documentsdb and vectorsdb.

Compatibility

  • Pre-2.0.0 clients can still send and receive the compound databaseId:collectionId selector.
  • Response formats below 1.7.0 continue removing resourceId, matching the pre-1.7 response contract.
  • Existing unresolved legacy records are preserved rather than guessed.
  • The dependency now uses the released utopia-php/migration ^1.18.0; no temporary migration VCS pin remains.

Validation

  • Focused migration/schema/filter/query suite: 23 tests, 203 assertions.
  • Formatter/linter: passed.
  • Full PHPStan: passed.
  • Composer validation and locked audit: passed, no advisories.
  • Full local unit run reached 940 tests; the remaining 10 errors and 2 failures require the unavailable local YAML/MaxMind extensions and signing-key fixture. GitHub CI is the environment-complete gate.

@greptile-apps

greptile-apps Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR exposes queryable resource links for migrations and keeps older migration clients compatible. The main changes are:

  • Adds source, parent, and destination resource fields to migration documents.
  • Indexes the new migration relation fields for queries.
  • Backfills legacy migration resource selectors when both generations can be resolved.
  • Splits legacy compound migration request IDs for pre-2.0 clients.
  • Rebuilds legacy migration response fields for older response formats.
  • Updates CSV and JSON migration creation and worker paths to use split resource IDs.
  • Bumps utopia-php/migration to the released 1.18.0 package.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
app/config/collections/projects.php Adds nullable migration relation fields and indexes for source, parent, and destination resource lookups.
app/controllers/general.php Registers the V27 request and response filters for clients using pre-2.0 formats.
composer.json Updates utopia-php/migration to the released 1.18.0 range.
composer.lock Locks utopia-php/migration to version 1.18.0.
src/Appwrite/Migration/Version/V25.php Backfills migration resource relation fields while preserving legacy records that cannot be safely resolved.
src/Appwrite/Platform/Workers/Migrations.php Routes worker source, destination, transfer, and usage-stat paths through resolved migration resource selectors.
src/Appwrite/Utopia/Request/Filters/V27.php Converts legacy compound migration resourceId request fields into split database and collection IDs.
src/Appwrite/Utopia/Response/Filters/V27.php Converts new migration resource relation fields back into the older response shape.
src/Appwrite/Platform/Modules/Migrations/Http/Migrations/CSV/Imports/Create.php Stores canonical resource metadata for CSV import migrations.
src/Appwrite/Platform/Modules/Migrations/Http/Migrations/CSV/Exports/Create.php Stores canonical resource metadata for CSV export migrations.
src/Appwrite/Platform/Modules/Migrations/Http/Migrations/JSON/Imports/Create.php Stores canonical resource metadata for JSON import migrations.
src/Appwrite/Platform/Modules/Migrations/Http/Migrations/JSON/Exports/Create.php Stores canonical resource metadata for JSON export migrations.

Reviews (23): Last reviewed commit: "Merge main: resolve composer conflicts, ..." | Re-trigger Greptile

Comment thread src/Appwrite/Utopia/Request/Filters/V25.php Outdated
Comment thread src/Appwrite/Migration/Version/V25.php Outdated
Comment thread src/Appwrite/Platform/Workers/Migrations.php Outdated
Comment thread src/Appwrite/Platform/Workers/Migrations.php
@github-actions

github-actions Bot commented May 1, 2026

Copy link
Copy Markdown

🔄 PHP-Retry Summary

Flaky tests detected across commits:

Commit 7f48a8a - 1 flaky test
Test Retries Total Time Details
LegacyCustomClientTest::testTimeout 1 139.25s Logs
Commit 4b5c123 - 1 flaky test
Test Retries Total Time Details
DocumentsDBCustomServerTest::testTimeout 1 128.35s Logs
Commit ea9a74b - 1 flaky test
Test Retries Total Time Details
DocumentsDBCustomServerTest::testTimeout 1 126.46s Logs
Commit 2d828af - 1 flaky test
Test Retries Total Time Details
FunctionsCustomServerTest::testDeleteExecution 1 149ms Logs
Commit 7d0e4ba - 1 flaky test
Test Retries Total Time Details
LegacyCustomClientTest::testNotEndsWith 1 240.89s Logs

Note: Flaky test results are tracked for the last 5 commits

@github-actions

github-actions Bot commented May 1, 2026

Copy link
Copy Markdown

✨ Benchmark results

Comparing main (before) → feat/migration-resource-attrs (after).

Metric Before After Change
🚀 Requests/sec 162.4 167.07 +2.9%
⏱️ Latency P50 87.58 ms 84.07 ms -4%
⏱️ Latency P95 313.66 ms 301.82 ms -3.8%
Per-scenario breakdown & investigation details

Metrics below reflect the current branch (after). Δ P95 compares against the base.

Scenario P50 (ms) P95 (ms) Requests RPS Δ P95 (ms)
API total 84.07 301.82 10,488 167.07 -11.84
Account 170.27 476.98 552 9.06 +57.8
TablesDB 80.27 266.88 5,704 91.91 -13.41
Storage 74.77 263.46 2,760 46.86 -36.44
Functions 131.91 415.84 1,472 25.59 +28.65

Top API waits (after)

API request Max wait (ms)
storage.files.download 1,001.86
account.name.update 943.65
account.get 831.16
tokens.delete 806.78
storage.files.view 803.8

Comment thread src/Appwrite/Utopia/Response/Filters/V25.php Outdated
@blacksmith-sh

blacksmith-sh Bot commented May 1, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
› Tests\E2E\Services\TablesDB\TablesDBCustomClientTest/
testEnforceCollectionPermissions
View Logs

Fix in Cursor

Copilot AI review requested due to automatic review settings June 10, 2026 05:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@ChiragAgg5k
ChiragAgg5k changed the base branch from 1.9.x to main June 23, 2026 09:53
@abnegate
abnegate force-pushed the feat/migration-resource-attrs branch from 7f48a8a to 2c02b5d Compare July 17, 2026 12:27
@abnegate abnegate changed the title refactor(migrations): split composite resourceId into parent/leaf feat(migrations): expose queryable resource links Jul 17, 2026
@abnegate
abnegate changed the base branch from main to 1.9.x July 17, 2026 12:27
@abnegate
abnegate force-pushed the feat/migration-resource-attrs branch from 2c02b5d to 4b5c123 Compare July 17, 2026 12:29
Comment thread src/Appwrite/Migration/Version/V25.php
@abnegate
abnegate changed the base branch from 1.9.x to main July 20, 2026 09:05
@abnegate
abnegate force-pushed the feat/migration-resource-attrs branch 2 times, most recently from 84490c6 to 9feaaff Compare July 20, 2026 10:01
Comment thread app/controllers/general.php
abnegate and others added 9 commits July 20, 2026 22:11
Fold the migration-resource-link schema provisioning and document
backfill into main's V25 migration instead of a standalone V25: only
the class mapped to the release version (2.0.0 => V25) is executed, so
a separate migration would never run. Main's existing V25 collection
work is preserved and the migrations-collection attributes/indexes plus
the resource-link document backfill are layered on top.

Gate the V27 request/response filters on < 2.0.0 (the release the fields
ship in) instead of the branch's 1.9.6, keeping the API version aligned
with the folded migration.

Restore main's utopia-php/auth VCS repository alongside the
utopia-php/migration one, keep utopia-php/messaging ^2.0, and regenerate
the lock so only utopia-php/migration moves to the resource-selector
branch. APP_VERSION_STABLE and the migration version map stay at main's
values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the dev-branch VCS pin now that feat/drop-compound-resource-id
shipped in the 1.18.0 tag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abnegate
abnegate force-pushed the feat/migration-resource-attrs branch from 9feaaff to aa9be97 Compare July 20, 2026 10:28
Comment thread app/controllers/general.php
Comment thread src/Appwrite/Migration/Version/V25.php
abnegate and others added 2 commits July 20, 2026 22:46
The release carrying this feature is 1.9.6, so the backfill moves out of
2.0.0's V25 into its own V26 migration keyed to 1.9.6, the V27
request/response filters gate at 1.9.6, and APP_VERSION_STABLE bumps to
match. V25 returns to main's contents.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abnegate
abnegate force-pushed the feat/migration-resource-attrs branch from a4483cf to 1544ad5 Compare July 20, 2026 13:29
abnegate and others added 3 commits July 21, 2026 01:45
CE main deliberately renamed the next release from 1.9.6 to 2.0.0
(923e413, 2026-07-02), so 1.9.6 will never ship. The feature stays
folded in V25 keyed to 2.0.0 with the V27 filters gated at 2.0.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…1.18.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abnegate
abnegate merged commit 7dfadfe into main Jul 20, 2026
78 of 79 checks passed
@abnegate
abnegate deleted the feat/migration-resource-attrs branch July 20, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants