fix(context): keep reruns emitted before the closing failure/error tag#1143
Merged
Merged
Conversation
Playwright's junit reporter (with includeRetries) emits a test that failed every attempt as a single <testcase> whose <rerunFailure>/<rerunError> attempts appear BEFORE the closing <failure>/<error>. The parser starts each testcase as Success, so those reruns are parsed into flaky_runs while the status is still Success. set_test_case_status then replaced the whole status enum with a fresh NonSuccess (empty reruns), discarding every earlier attempt — so a fail-all-retries test collapsed to a single run and pass-on-retry never surfaced for the failing case. Carry the accumulated reruns onto the terminal status (mem::take the flaky_runs and add_reruns them to the new failure/error) so each attempt surfaces as its own run. The pass-on-retry (flaky) path was unaffected because it has no closing <failure> to clobber the runs. Adds a regression test using Playwright's real element order (system-out -> rerunFailure... -> failure); the existing failing-rerun test only passed because it ordered <failure> before <rerunFailure>.
|
😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details. |
…Playwright The original #1142 test ordered <failure> before <rerunFailure>, and real Playwright output is the reverse. Rework the regression test to mirror the actual reporter output for a test that failed every attempt: reruns emitted BEFORE the terminal element, a mix of <rerunFailure> (assertion) and <rerunError> (timeout), and a terminal <error> (attempt 0 timed out) rather than <failure>. Asserts all four attempts surface as their own runs with each attempt's message preserved. Verified it fails without the fix.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1143 +/- ##
==========================================
+ Coverage 82.75% 83.05% +0.29%
==========================================
Files 70 70
Lines 15788 15850 +62
==========================================
+ Hits 13066 13164 +98
+ Misses 2722 2686 -36 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
dfrankland
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the case where a test that fails every retry is uploaded as a single run instead of one run per attempt — the failing-side counterpart to #1142 (TRUNK-18796).
Playwright's JUnit reporter (with
includeRetries) emits a fail-all-retries test as a single<testcase>whose<rerunFailure>/<rerunError>attempts appear before the closing<failure>/<error>:open_test_casestarts each testcase asTestCaseStatus::Success, so each<rerunFailure>parsed before the<failure>is filed underflaky_runs(quick-junit routes reruns toflaky_runswhile the status isSuccess). When the trailing<failure>arrives,set_test_case_statusreplaced the entire status enum with a freshNonSuccess { reruns: [] }, discarding every earlier attempt. Result: the test collapses to a single run, and the retries survive only as concatenated text in the failure message.The pass-on-retry (flaky) path was unaffected because a passing testcase has no closing
<failure>to clobber the accumulated runs — which is exactly why pass-on-retry surfaced multiple runs but fail-all-retries did not.Fix
In
set_test_case_status, before overwriting aSuccessstatus,mem::takethe accumulatedflaky_runsandadd_rerunsthem onto the new failure/error status, so each attempt still surfaces as its own run.Test
Adds
test_into_test_case_runs_emits_run_per_failing_rerun_playwright_order, which uses Playwright's real element order (system-out→rerunFailure… →failure). The existingtest_into_test_case_runs_emits_run_per_failing_rerunpassed only because it ordered<failure>before<rerunFailure>— the reverse of Playwright's actual output — which is why the regression slipped through.contextparser suite green (15/15), including the existing Surface individual rerun attempts as separate test runs #1142 tests.Notes
internal.binfor a fail-all-retries run carried only the attempt-0 attachment in structuredsystem_out(retries only in text), which pinpointed the dropped<rerunFailure>elements.🤖 Generated during triage of the 0.15.0-beta.1 rerun report.