Skip to content

fix(context): keep reruns emitted before the closing failure/error tag#1143

Merged
trunk-io[bot] merged 2 commits into
mainfrom
claude/fix-junit-rerun-drop-before-failure
Jul 15, 2026
Merged

fix(context): keep reruns emitted before the closing failure/error tag#1143
trunk-io[bot] merged 2 commits into
mainfrom
claude/fix-junit-rerun-drop-before-failure

Conversation

@mmatheson

Copy link
Copy Markdown
Member

Summary

Fixes the case where a test that fails every retry is uploaded as a single run instead of one run per attempt — the failing-side counterpart to #1142 (TRUNK-18796).

Playwright's JUnit reporter (with includeRetries) emits a fail-all-retries test as a single <testcase> whose <rerunFailure>/<rerunError> attempts appear before the closing <failure>/<error>:

<testcase name="..." >
  <system-out>...</system-out>        <!-- attempt 0 only -->
  <rerunFailure .../>                 <!-- retry 1 -->
  <rerunFailure .../>                 <!-- retry 2 -->
  <failure .../>                      <!-- appended last -->
</testcase>

open_test_case starts each testcase as TestCaseStatus::Success, so each <rerunFailure> parsed before the <failure> is filed under flaky_runs (quick-junit routes reruns to flaky_runs while the status is Success). When the trailing <failure> arrives, set_test_case_status replaced the entire status enum with a fresh NonSuccess { reruns: [] }, discarding every earlier attempt. Result: the test collapses to a single run, and the retries survive only as concatenated text in the failure message.

The pass-on-retry (flaky) path was unaffected because a passing testcase has no closing <failure> to clobber the accumulated runs — which is exactly why pass-on-retry surfaced multiple runs but fail-all-retries did not.

Fix

In set_test_case_status, before overwriting a Success status, mem::take the accumulated flaky_runs and add_reruns them onto the new failure/error status, so each attempt still surfaces as its own run.

Test

Adds test_into_test_case_runs_emits_run_per_failing_rerun_playwright_order, which uses Playwright's real element order (system-outrerunFailure… → failure). The existing test_into_test_case_runs_emits_run_per_failing_rerun passed only because it ordered <failure> before <rerunFailure> — the reverse of Playwright's actual output — which is why the regression slipped through.

Notes

  • Client-side only — the parser runs in the customer's CI and drops the attempts before upload, so ingestion/portal/monitors were never at fault. Bundles already uploaded by the buggy CLI stay one-run (the attempts were never captured).
  • Verified against a real staging bundle: the internal.bin for a fail-all-retries run carried only the attempt-0 attachment in structured system_out (retries only in text), which pinpointed the dropped <rerunFailure> elements.

🤖 Generated during triage of the 0.15.0-beta.1 rerun report.

Playwright's junit reporter (with includeRetries) emits a test that failed
every attempt as a single <testcase> whose <rerunFailure>/<rerunError> attempts
appear BEFORE the closing <failure>/<error>. The parser starts each testcase as
Success, so those reruns are parsed into flaky_runs while the status is still
Success. set_test_case_status then replaced the whole status enum with a fresh
NonSuccess (empty reruns), discarding every earlier attempt — so a
fail-all-retries test collapsed to a single run and pass-on-retry never surfaced
for the failing case.

Carry the accumulated reruns onto the terminal status (mem::take the flaky_runs
and add_reruns them to the new failure/error) so each attempt surfaces as its
own run. The pass-on-retry (flaky) path was unaffected because it has no closing
<failure> to clobber the runs.

Adds a regression test using Playwright's real element order (system-out ->
rerunFailure... -> failure); the existing failing-rerun test only passed because
it ordered <failure> before <rerunFailure>.
@trunk-io

trunk-io Bot commented Jul 15, 2026

Copy link
Copy Markdown

😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details.

…Playwright

The original #1142 test ordered <failure> before <rerunFailure>, and real
Playwright output is the reverse. Rework the regression test to mirror the
actual reporter output for a test that failed every attempt: reruns emitted
BEFORE the terminal element, a mix of <rerunFailure> (assertion) and
<rerunError> (timeout), and a terminal <error> (attempt 0 timed out) rather
than <failure>. Asserts all four attempts surface as their own runs with each
attempt's message preserved. Verified it fails without the fix.
@TylerJang27 TylerJang27 changed the title fix(context): keep reruns emitted before the closing &lt;failure&gt;/&lt;error&gt; fix(context): keep reruns emitted before the closing failure/error tag Jul 15, 2026
@TylerJang27
TylerJang27 requested a review from dfrankland July 15, 2026 19:03
@codecov-commenter

codecov-commenter commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.38710% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.05%. Comparing base (7704b9d) to head (030e508).

Files with missing lines Patch % Lines
context/src/junit/parser.rs 98.38% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1143      +/-   ##
==========================================
+ Coverage   82.75%   83.05%   +0.29%     
==========================================
  Files          70       70              
  Lines       15788    15850      +62     
==========================================
+ Hits        13066    13164      +98     
+ Misses       2722     2686      -36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trunk-staging-io

trunk-staging-io Bot commented Jul 15, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

Failed Test Failure Summary Logs
pending_quarantine_test should be quarantined when run with variant A test marked as pending was expected to fail but unexpectedly passed. Logs ↗︎
variant_quarantine_test should be quarantined when run with variant A test expected the sum of 2 + 2 to be 5, but it was actually 4, indicating a failing assertion. Logs ↗︎

View Full Report ↗︎Docs

@trunk-io

trunk-io Bot commented Jul 15, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@trunk-io
trunk-io Bot merged commit 3fb8713 into main Jul 15, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants