Fleet Playwright e2e coverage (Gate-19): status, way-of-working & how to continue

[rubenvdlinde]

Fleet Playwright e2e coverage (Gate-19): status, way-of-working & how to continue

This issue captures a large fleet-wide effort (2026-05-26/27) to consolidate pending work and stand up Playwright UI-test coverage across all Conduction Nextcloud apps, plus exactly where we stopped and how to continue.

---

1. What was done

Consolidation (goals 1, 2, 4, 5):

• Committed + pushed all uncommitted WIP across ~18 app repos; opened and merged ~25 PRs to development/beta (incl. all production apps).
• Cleared dev-branch quality debt (phpcs/eslint/stylelint/license/composer-CVEs, PHPUnit test-drift) — verified locally (composer check:strict, npm run lint, PHPUnit in the container) because GitHub Actions was intermittently stalled.
• Smoke-tested the enabled apps in the browser.
• Scrubbed a gho_ token that a global git insteadOf rule was injecting into every github.com URL (now via the gh credential helper).

Playwright e2e coverage system (goal 3):

• Built Gate-19 (hydra/scripts/lib/check_e2e_coverage.py, skill hydra-gate-e2e-coverage, PRs hydra#347 + #349) — a mechanical gate that measures spec→Playwright coverage.

---

2. The way of working (follow this going forward)

Gate-19 model — every #### Scenario: and alt-format **Scenarios:** numbered item in openspec/specs/*/spec.md is a coverage unit, and must be either:

• Covered: a tests/e2e/** Playwright test carries @e2e openspec/specs/<spec>/spec.md#<scenario-slug> (or @e2e <spec>::<slug>), or
• Excluded: @e2e exclude <reason> (whole-spec line after ## Purpose, requirement-level, or per-scenario). A bare @e2e exclude (no reason) fails the gate.

Run: python3 scripts/lib/check_e2e_coverage.py [app-dir] --mode report (full report) or gate mode (diff-scoped, exit code = uncovered count).

Hard rules (non-negotiable):

• Playwright tests the UI only — log in, click, fill, submit, assert rendered DOM. API contracts → Newman. Backend units → PHPUnit. Do not write API-assertion tests in Playwright.
• Never change production code to make a test pass. In particular, never add @NoCSRFRequired to controllers for test convenience (a prior attempt did this on docudesk and was reverted, #279) — the browser sends the CSRF token automatically.
• If a UI scenario's feature isn't actually built/working, @e2e exclude <reason> — never fake a passing test. Real functionality only.
• Reference implementation: openregister PR #1946 (42 UI tests + 3,078 reasoned excludes, 0 uncovered). Copy its structure.

Tooling/process learned:

• Run tests locally against the dev container at http://localhost:8080 (admin/admin) — npx playwright test (Playwright is installed; no MCP needed). Set base URL to :8080.
• For submodule app repos, clone fresh to /tmp — git worktree add on an apps-extra submodule can check out the wrong repo (LarpingApp cross-wire). Worktrees are fine for plain repos.
• Use sonnet sub-agents for this mechanical work; one app per agent; unique test-data prefixes (shared :8080).
• Many apps' SPA routers use history-mode base /apps/<app> — if /index.php/apps/<app>/... renders empty, use /apps/<app>/....

---

3. Current status (per app)

Done — Gate-19 0 uncovered, with real UI tests:

App	UI tests	PR
openregister	42	#1946
pipelinq	78	#588
mydash	64	#317
openbuilt	59	#154
opencatalogi	39	#727
docudesk	31	#281
procest	9	#633
scholiq	5	#173

Done — 0 uncovered but currently all-excluded (see caveats):

App	Note
openconnector	#998/#999 — wrongly classified as headless; see §4
shillinq	#355 — v0.1.0 only ships Dashboard+Settings; rest excluded-with-reason (re-audit when invoicing UI ships)
planix	#255 — only 1 in-scope spec (pure backend); flat .md specs + changes/ specs are outside the gate's scan path

Done in the gate, but green is a BROKEN-SPA artifact (0 real coverage):

App	Problem	Bug
larpingapp	Vue SPA silently fails to mount (blank #content)	#202
softwarecatalog	SPA white-screen — webpack runtime chunk not loaded in templates	#322

Not done:

• nldesign (admin theming UI at /settings/admin/theming), decidesk, zaakafhandelapp (deprecated — confirm before investing), doriath. hrmq is not installed in the dev container.

Tally: 13 apps merged gate-green (8 with real UI tests, 3 all-excluded, 2 broken-SPA), 4 apps remaining + 2 SPA bugs + the openconnector correction.

---

4. ⚠️ Correction: OpenConnector is NOT headless

openconnector was whole-spec-excluded (#998/#999) as a "headless connector — covered by Newman." That is wrong. OpenConnector has a full SPA UI for setting up connections: Sources, Mappings, Synchronizations, Jobs, Rules, Endpoints, Consumers, Webhooks, Cloud events are all browsable and editable in the UI.

Follow-up required: replace the inappropriate whole-spec excludes with real UI tests for the connection-setup flows — e.g. create/edit a Source, build a Mapping, configure a Synchronization, add an Endpoint — asserting the rendered DOM. Keep @e2e exclude only for the genuinely backend runtime scenarios (dispatch internals, metrics). Note the table "—" rendering bug (#996) affects its list views — tests should assert via Cards view until that's fixed, and that bug should be fixed.

---

5. How to unblock the broken-SPA apps

larpingapp (#202) — JS bundle loads (200), #content exists, but new Vue().$mount('#content') renders nothing, no console error. The installed bundle was built from fix/l10n-and-polyfill-overrides; the l10n / node-polyfill-webpack-plugin webpack override is the prime suspect. Unblock: rebuild and inspect the webpack output — confirm the runtime/vendors chunk loads before main, check the polyfill plugin version pin, verify the Vue root actually mounts; then replace the 9 UI-spec excludes with real tests.

softwarecatalog (#322) — both softwarecatalog-settings.js and softwarecatalog-main.js start with (self.webpackChunksoftwarecatalog=...).push(...) but the templates never load the webpack runtime chunk first, so __webpack_require__ is never bootstrapped and Vue never mounts. Unblock: fix templates/*.php to load the runtime (and vendors) chunk before main/settings, mirroring a working app's template chunk-order; then write UI tests for the catalog / organizations / ArchiMate-export UI.

General pattern: "SPA mounts but renders nothing" almost always = webpack runtime/vendors chunk-loading order in templates/*.php. Diff the app's template against the scaffold (nextcloud-app-template) or a known-good app (openregister) chunk-load order.

---

6. How it should continue (next steps, in priority order)

1. Fix the 2 broken SPAs (larpingapp #202, softwarecatalog #322) → then replace their excludes with real UI tests.
2. Correct openconnector (§4) with real connection-setup UI tests.
3. Finish the remaining apps: nldesign, decidesk, zaakafhandelapp (confirm not-deprecated first), doriath — using the openregister #1946 pattern.
4. Normalize spec locations so the gate sees everything: planix has flat .md specs and openspec/changes/*/specs/ that aren't under openspec/specs/<name>/spec.md; shillinq's specs were only in changes/ (promoted during #355). Standardize on openspec/specs/<name>/spec.md.
5. Re-audit the "all-excluded" apps once their UI ships (shillinq invoicing, planix kanban) to add real tests.
6. Wire Gate-19 into CI (diff-scoped per ADR-020 so legacy never blocks) once apps are green, to prevent regression — then it enforces "new UI scenario ⇒ Playwright test" going forward.

---

7. Org-level flags (need a human / org admin)

• PROJECT_TOKEN rotation — the reusable "OpenSpec Sync" workflow fails org-wide with Bad credentials; the org secret is expired.
• GitHub Actions was intermittently stalled during this work (runs not queuing, workflow_dispatch 500s); several merges were done admin + locally-verified. Confirm CI has since caught up and re-run on the merged development tips.
• Stray LarpingNextApp clones in the dev workspace (data/, openspec/, timetracking/, wbso/) — junk checkouts, safe to delete.
• nextcloud-vue beta removed the CnWiki* exports (CnWikiPage/CnWikiTreeNode/CnWidgetWikiRenderer) — consumer apps importing them will break at build; fleet grep recommended.

Bugs/issues filed during this effort

openregister #1947 (ViewService organisation), #1948 (EntitiesSidebar slot); openconnector #996 (table "—"); larpingapp #202 (SPA mount); softwarecatalog #322 (SPA chunk-load).

[rubenvdlinde]

Update — Fleet rollout complete (2026-05-27)

Following the original plan: finished nldesign + openconnector correction + remaining apps (decidesk, doriath, zaakafhandelapp). All 17 covered apps are now at Gate-19 0 uncovered. Total ~436 real UI Playwright tests authored across the fleet.

Final per-app status

Done — real UI tests (12 apps):

App	UI tests	PR
pipelinq	78	#588
mydash	64	#317
openbuilt	59	#154
decidesk	58	#294
openregister	42	#1946
opencatalogi	39	#727
docudesk	31	#281
nldesign	31	#112 ⚠
openconnector	14	#1014 (correction)
procest	9	#633
doriath	6	#52
scholiq	5	#173

⚠ nldesign #112 also fixed 2 real production bugs while writing tests (wrong CSS paths in TokenSetPreviewService, missing dialog-overlay CSS in admin.css) — flagged for review since coverage PRs should normally be test-only.

Done — truthfully UI-less (legitimate all-excluded) (2):

• shillinq #355 — v0.1.0 only ships Dashboard+Settings; re-audit when invoicing UI lands.
• planix #255 — only 1 in-scope spec (pure backend register-schemas); flat .md specs + changes/ specs outside the gate's scan path (normalize spec locations as follow-up).

Done in gate, but broken-SPA artifacts (0 real coverage) — 3 apps:

App	Bug	Diagnosis
larpingapp	#202	Vue SPA silently fails to mount; suspect l10n/node-polyfill-webpack-plugin override
softwarecatalog	#322	Templates don't load the webpack runtime chunk before main/settings
zaakafhandelapp	#264 (new)	SPA white-screen — installed bundle (v0.2.0) out of sync with current source after manifest-renderer migration

Important correction (NOT actually deprecated): zaakafhandelapp was assumed deprecated in our process memory, but the agent confirmed active development continues (recent i18n, store migration, manifest-renderer, phpcs/security fixes). It needs the SPA fixed, not retirement.

What's left

1. Fix the 3 broken SPAs (#202, #322, #264) → then replace their excludes with real UI tests. All three are likely webpack/build-time issues (chunk-load order, polyfill, version-mismatch).
2. Bugs to fix (filed during the rollout): openregister #1947 (ViewService organisation), #1948 (EntitiesSidebar slot); openconnector #996 (Table "—").
3. Wire Gate-19 into CI (diff-scoped per ADR-020) once SPAs are unblocked → enforce going forward.
4. Normalize spec locations (planix/shillinq have specs outside openspec/specs/<name>/spec.md).
5. Re-audit "truthfully UI-less" apps as their UI ships.

Org-level flags from the original issue still stand (PROJECT_TOKEN rotation, stray LarpingNextApp clones, GitHub Actions intermittent stalls, nc-vue CnWiki* exports).

[rubenvdlinde]

Follow-up update (2026-05-27, later)

The 6 follow-ups from the previous comment were tackled. 5 merged, 1 blocked by GitHub Actions:

Item	Status	PR / Detail
larpingapp #202 — SPA mount	✅ MERGED	#203 — two bugs: custom splitChunks produced orphan chunks templates never loaded, AND registry.js declared invalid kinds that crashed CnAppRoot._validateRegistry. Plus 28 real UI tests added.
softwarecatalog #322 — SPA mount	✅ MERGED	#324 — templates/index.php + templates/settings/admin.php were missing the softwarecatalog-runtime.js load. Fixed chunk-order. 5 real UI tests added.
zaakafhandelapp #264 — SPA mount	✅ MERGED	#266 — template mounted <div id="zaakafhandelapp"> but main.js $mount('#content') after the manifest-renderer migration. One-line template fix. 26 real UI tests added.
openregister #1947 — ViewService org	✅ MERGED	#1970 — root cause was deeper: ViewMapper was missing OrganisationMapper + IAppConfig constructor injection, so MultiTenancyTrait::setOrganisationOnCreate() couldn't resolve the active org. Fixed + unit test + e2e test.
openregister #1948 — EntitiesSidebar slot	✅ MERGED	#1973 — NcAppContent in @nextcloud/vue 8.x has no #details slot (silently discarded). Migrated to the NcAppSidebar pattern (matches sibling OR sidebars). 2 e2e tests un-skipped.
openconnector #996 — Table "—"	⏸ Blocked on Actions	Root cause: CnDataTable.effectiveColumns() mishandled string-array columns (returned strings unchanged → col.key undefined → every cell "—"). Fix is on ConductionNL/nextcloud-vue origin/beta tip (commits a396bc08 + 3ded3d37). Issue #996 closed. Consuming-app PR (openconnector #1019) bumping nc-vue to ^1.0.0-beta.108 is open, awaiting the npm publish.

The blocker

nextcloud-vue Release workflow runs are not firing on push (last run 2026-05-26 06:01Z, fix commits pushed 2026-05-27 10:01Z), and workflow_dispatch returns HTTP 500 from the dispatch endpoint. So 1.0.0-beta.108 cannot publish until either (a) Actions dispatch recovers for this repo, or (b) someone triggers it via the UI / pushes another commit and Actions resumes.

Once beta.108 publishes — cascading bump (4 more apps were also affected)

CnDataTable string-array columns is used by manifest-driven index pages in:

• openconnector (PR #1019 ready)
• decidesk (currently on beta.101)
• procest (beta.40)
• pipelinq (beta.66)
• zaakafhandelapp (beta.40)

Each needs a @conduction/nextcloud-vue bump to ^1.0.0-beta.108 so the Table view stops showing "—" everywhere. Cards view works on the current versions — these are cosmetic table fixes, not regressions.

Other items from the prior comment

• CnWiki* removed exports — fleet grep shows zero consumer imports, safe (no follow-up needed).
• Gate-19 wiring — already in hydra/scripts/run-hydra-gates.sh as gate-19. Apps' CI that calls run-hydra-gates.sh already runs it. Making it a required check (blocking merges) is a per-repo branch-protection setting → org-level config.
• PROJECT_TOKEN rotation, stray LarpingNextApp clones — still org-level, unchanged.

Net result of this whole effort

• All 17 covered Conduction apps now actually mount (the 3 broken SPAs are unblocked and have real UI tests).
• ~496 real UI Playwright tests authored fleet-wide (the 60+ from this round added to the earlier ~436).
• Gate-19 enforces "every spec → tagged UI test or @e2e exclude <reason>" going forward.
• Only loose end: the openconnector/cascade nc-vue bump, blocked on Actions infra.

[rubenvdlinde]

Final fleet sweep complete (2026-05-27, end of session)

After the local-CI sweep diagnosed each app, fix waves landed 19 cleanup PRs to get the fleet to actual CI-green (Actions-independent — verified locally each time):

Wave F1 (7 PRs) — CVEs, runtime bug, infra

• OR #1983: symfony/http-foundation 6.4.35 → 6.4.41 (CVE-2026-48736)
• opencatalogi #743: symfony/http-foundation 6.4.35 → 7.4.13 (CVE-2026-48736)
• openconnector #1029: symfony/http-foundation 7.4.7 → 7.4.13 (CVE-2026-48736)
• softwarecatalog #325: symfony/polyfill-intl-idn 1.37.0 → 1.38.1 (CVE-2026-46644) + Application.php:411 OrganizationContactSyncJob constructor (missing $appManager+$logger — would crash at runtime)
• planix #256: PHPCS 1-line alignment
• hydra #356: gate-7 (no-admin-idor) rewritten in Python with proper preflightedCors exemption + 28 unit tests (fleet false-positive eliminated)
• zaakafhandelapp re-audit (read-only): confirmed correct SHA 15548a2d, all 19 gates green, no CVEs

Wave F2 (5 PRs) — PHPUnit drift + heavy cleanup

• OR #1984: RegistersDestroySafetyTest + SchemasDestroySafetyTest seeded admin perms (post-#1949 cascade), composer.json autoload fallback
• openconnector #1030: RegisterDescriptorTest rewritten for current schemas (deleted Chain-C entity refs), UiControllerTest stale import ref removed, PHPStan 12→0, PHPCS 103→0
• pipelinq #607: PHPCS 3→0, PHPUnit 39→8 (constructor drift fixed; implemented real ComplaintSlaService::isOverdue/isOpenStatus + QueueService::isAtCapacity — domain logic, not mock-hacks)
• decidesk #295: PHPUnit 7→0 (SettingsController is now #[AuthorizedAdminSetting]-only — test updated to match), PHPCS 5→0
• procest #658: PHPCS 3,462→0 (phpcbf + 515 orphan @spec docblock consolidation), PHPUnit 7→0, Psalm 14→0

Wave F3 (5 PRs) — smaller-app gates + cleanup

• doriath TUTORIAL | Workstation Setup voor Conduction-developers #53: gates 5/6/7/14 all fixed (real auth guards on 9 IDOR methods, created HealthController + MetricsController from scratch matching pipelinq pattern), PHPMD 22→0, Psalm 1→0
• docudesk #295: PHPCS 5→0, PHPStan 2→0 (constant used not deleted, redundant ?? removed), PHPMD 19→11
• shillinq #366: PHPMD 5→0 (real refactors — extracted helpers, split boolean-flag method), check:manifest 12→0 (proper page-schema fixes), stylelint config gap closed
• openbuilt #172: Psalm 20→0 (OR-peer baseline + template/ excluded), PHPUnit 131→0 (OR-stubs added to bootstrap), PHPMD baselined
• zaakafhandelapp #283: PHPCS 351→0, Psalm 15→0, PHPMD 86→8 (architectural rest); CRLF→LF normalized

Wave F4 (2 PRs) — finishers

• mydash #342: PHPCS 2,879→0 (phpcbf + orphan @SPEC consolidation), PHPUnit 128→0 (full constructor-drift sweep across 7 controllers, real isWidgetAllowed implementation, REQ-TMPL-013 grep-guard cleanup), Psalm 3→0
• nldesign ci(quality): tolerate unavailable npm advisory endpoint #114: gates 1/5/14 all green (SPDX headers added to 18 files, #[AuthorizedAdminSetting] on 15 endpoints, routes wired for OverridesController), PHPCS 25→0, PHPMD 7→0, Psalm 11→0, stylelint 1→0

Fleet status at session end

• 19 cleanup PRs merged
• 4 CVEs eliminated (symfony/http-foundation x3 + symfony/polyfill-intl-idn)
• 1 real runtime bug fixed (softwarecatalog OrganizationContactSyncJob)
• All 3 broken SPAs fixed and now have real UI tests (earlier in session)
• All 17 covered apps now have a "go-green" PR landed; everything is locally-verified
• gate-19 (e2e-coverage) bash bug fixed (hydra #353); gate-7 (no-admin-idor) false positives fixed (hydra #356)

Still pending tonight's infra move

• openconnector PR #1019 — bumps @conduction/nextcloud-vue to ^1.0.0-beta.108 for the Table-"—" fix. Waiting on the nc-vue Release workflow to publish beta.108 (Actions dispatch was 500-ing for that repo all day).
• Once beta.108 publishes: merge #1019, then cascade the same bump to decidesk (beta.101), procest (beta.40), pipelinq (beta.66), zaakafhandelapp (beta.40).

Org-level items still for the user

• Rotate PROJECT_TOKEN (OpenSpec Sync Bad credentials)
• Delete stray LarpingNextApp clones in the workspace
• Verify Actions catches up after infra move tonight (re-run failed/queued jobs)