Purge (micro)schema versions by plyhun · Pull Request #1753 · gentics/mesh

plyhun · 2026-03-24T15:45:39Z

Abstract

To same database time/space, we could delete empty versions.

Checklist

General

Added abstract that describes the change
Added changelog entry to /CHANGELOG.adoc
Ensured that the change is covered by tests
Ensured that the change is documented in the docs

On API Changes

Checked if the changes are breaking or not
Added GraphQL API if applicable
Added Elasticsearch mapping if applicable

plyhun · 2026-03-24T15:47:56Z

@@ -1,140 +0,0 @@
-package com.gentics.mesh.core.project.maintenance;


Completely unused. The correct implementation of ProjectVersionPurgeHandler is loaded elsewhere.

plyhun · 2026-03-24T15:49:02Z

 		// truncate the error detail message to the max length for the error detail property
 		setErrorDetail(truncateStackTrace(stackTrace));
-		setErrorMessage(e.getMessage());
+		setErrorMessage(StringUtils.substring(e.getMessage(), 0, 255));


DB limits overflow.

plyhun · 2026-03-24T15:50:43Z

 		}
 		if (version.getPreviousVersion() != null) {
-			version.getPreviousVersion().setNextVersion(null);
+			version.getPreviousVersion().setNextVersion(version.getNextVersion());


Bug in version removal. Worth porting to hotfix.

plyhun · 2026-03-24T15:51:29Z

 				if (status != null) {
 					boolean allMatching = true;
 					for (JobResponse info : response.getData()) {
+						if (before.getData().stream().anyMatch(j -> Objects.equal(j.getUuid(), info.getUuid()))) {


We count only new jobs.

plyhun · 2026-03-24T15:53:19Z

+
+	@Parameters(name = "{index}: {0}")
+	public static Collection<Object[]> parameters() {
+		return List.of(new Object[] { 0 }, new Object[] { 1 }, new Object[] { 2 }, new Object[] { 5 }, new Object[] { 8 });


Random meaningful numbers, including 1 (nothing to clean) and 0 (no changes).

plyhun · 2026-03-24T15:53:36Z

 	}

 	@Test
-	@Category({FailingTests.class})


Not anymore failing.

Updated the changelog entry to specify that the purge process saves storage resources.

npomaroli · 2026-03-30T12:36:53Z

+	 * @param version
+	 * @return
+	 */
+	Stream<? extends HibNodeFieldContainer> getFieldsContainers(HibSchemaVersion version);


Reading all contents for a schema version is extremely inefficient. Also this method is apparently not used anywhere, so better remove it.

npomaroli · 2026-03-30T12:38:40Z

+	 * @param version
+	 * @return
+	 */
+	Stream<? extends HibMicronode> getFieldsContainers(HibMicroschemaVersion version);		


Same comment as above. This method is used from PersistingMicroschemaDao.delete(HibMicroschema), but only to check, whether there is still a micronode using a specific version (before deleting it). There are better methods to check this.

npomaroli · 2026-03-30T12:46:12Z

+					.map(version -> version.getUuid())
+					.collect(Collectors.toSet());
+
+			containerDao.findAll().stream()


This seems to be inefficient and to produce long lasting db locks.
I tested with a database containing a schema with 22.000 versions. The job was running for several minutes until I gave up (there is no visible progress information). During that time, some requests (like getting the schemas for a specific branch) were hanging.

npomaroli · 2026-03-30T12:57:27Z

 			crudHandler.handleDelete(ac, uuid);
 		}, isOrderedBlockingHandlers());
+
+		InternalEndpointRoute purgeVersionsEndpoint = createRoute();


Using the endpoint DELETE /api/v2/schemas for purging unused schema versions is counterintuitive (same for micoschemas).

…terized requests.

Updated the purge process for schemas and microschemas to use new REST API endpoints, allowing for parameterization by schema name/UUID filters.

plyhun · 2026-04-01T08:44:22Z

@@ -0,0 +1,2 @@
+Core: Now it is possible to purge empty (micro)schema versions automatically, via new background jobs. This will save storage and runtime resources of the database.
+ The purge process is triggered via REST API `POST /utilities/purge/schemas/versions` and `POST /utilities/purge/microschemas/versions` respectively, and can be parameterized by schema name/UUID filters.


The calls have to be moved to /utilities because of POST /schemas/{uuid} taking over the /schemas/purge calls

plyhun · 2026-04-01T08:45:48Z

-		endpoint.exampleRequest(microschemaExamples.getGeolocationMicroschemaCreateRequest());
-		endpoint.exampleResponse(OK, utilityExamples.createValidationResponse(), "The validation report");
-		endpoint.blockingHandler(rc -> {
+	private void addValidationHandler() {


Endpoints grouped.

plyhun · 2026-04-01T08:46:53Z

+	/**
+	 * Set the job start query.
+	 */
+	void setQuery(String query);


General purpose job query.

plyhun · 2026-04-02T07:52:16Z

+		purgeSchemaVersionsEndpoint.method(POST);
+		purgeSchemaVersionsEndpoint.description("Purge the unused schema versions.");
+		purgeSchemaVersionsEndpoint.consumes(APPLICATION_JSON);
+		purgeSchemaVersionsEndpoint.exampleRequest(miscExamples.createNameOrUuidsRequest());


I favored a POST body over query parameters request, as it can hold more values, if required. Matter of a discussion though.

plyhun · 2026-04-20T13:51:23Z

-
 	@Override
 	public void beforeVersionDeletedFromDatabase(HibMicroschemaVersion version) {
+		HibernateTx tx = HibernateTx.get();


It really looks to me that no one had ever tested the deletion of microschema versions until this moment.

plyhun added 7 commits March 19, 2026 15:34

GPU-2209: Schema version purge

600f89f

Presumably works

6b9f1e9

Log the progress

377d0fb

Changelog

7473223

Fixes

b0be13d

Schema test

02dc08e

More tests

65117ce

plyhun requested a review from npomaroli March 24, 2026 15:45

plyhun self-assigned this Mar 24, 2026

plyhun commented Mar 24, 2026

View reviewed changes

Clarify storage savings in schema purge feature

8d85ad5

Updated the changelog entry to specify that the purge process saves storage resources.

npomaroli requested changes Mar 30, 2026

View reviewed changes

plyhun and others added 3 commits April 1, 2026 09:59

Transaction per version. Endpoints moved to /utilities + POST. Parame…

2f9dc2b

…terized requests.

Revise purge process for schemas and microschemas

74a5b95

Updated the purge process for schemas and microschemas to use new REST API endpoints, allowing for parameterization by schema name/UUID filters.

Simplify the endpoint. Docs.

a135e9c

plyhun commented Apr 1, 2026

View reviewed changes

Cleanup

8ab6016

plyhun requested a review from npomaroli April 1, 2026 08:52

Separate client methods with nullable bodies

68f294d

plyhun commented Apr 2, 2026

View reviewed changes

npomaroli and others added 3 commits April 13, 2026 15:46

Improve performance for schema version purge

9e7899f

Performance

080f591

Cleanup

7e38bfd

plyhun added 2 commits April 17, 2026 19:08

Performance v2

ba3f525

Sparse versions tests. Edge case fix.

8312d28

plyhun commented Apr 20, 2026

View reviewed changes

Merge remote-tracking branch 'origin/dev' into f-gpu-2209

50bac30

		@@ -1,140 +0,0 @@
		package com.gentics.mesh.core.project.maintenance;

		@@ -0,0 +1,2 @@
		Core: Now it is possible to purge empty (micro)schema versions automatically, via new background jobs. This will save storage and runtime resources of the database.
		The purge process is triggered via REST API `POST /utilities/purge/schemas/versions` and `POST /utilities/purge/microschemas/versions` respectively, and can be parameterized by schema name/UUID filters.

Uh oh!

Conversation

plyhun commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Abstract

Checklist

General

On API Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plyhun Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

plyhun commented Mar 24, 2026 •

edited

Loading

plyhun Mar 24, 2026 •

edited

Loading