Surface individual rerun attempts as separate test runs by EliSchleifer · Pull Request #1142 · trunk-io/analytics-cli

EliSchleifer · 2026-07-13T22:53:17Z

Summary

This change enables the JUnit parser to emit individual rerun/flaky attempts as separate test case runs, allowing failed attempts that eventually pass (or fail again) to be properly surfaced and classified as flaky tests.

Key Changes

Extract reruns from test case status: Added logic to collect TestRerun entries from both Success (flaky_runs) and NonSuccess (reruns) status variants
New helper function test_case_run_for_rerun(): Creates a TestCaseRun for each prior rerun attempt by:
- Cloning the base test run to inherit identity (test id, file, codeowners, quarantine state)
- Setting status to Failure to record the attempt as failed
- Populating output fields (message, description, system_out, system_err) from the rerun data
- Preserving timing information (timestamp and duration) from the rerun
Emit reruns before final attempt: Modified the test case run collection to push rerun attempts before the final run, ensuring chronological ordering
Added comprehensive test coverage: Two new tests validate:
- Pass-on-retry flakes (Playwright <flakyError> elements) emit both the failed attempt and final passing run
- Consistently failing tests emit each <rerunFailure> attempt plus the final failure

Implementation Details

Reruns are only emitted if they exist; skipped tests produce no reruns
Each rerun inherits all identity information from the final attempt to maintain test correlation
Timing data is carefully reconstructed from rerun timestamps and durations
The approach preserves backward compatibility while enabling flaky test detection

https://claude.ai/code/session_017d3M95mohe3LhktThN2rxz

into_test_case_runs() walked each <testcase> once and pattern-matched the status with `..`, discarding the parsed flaky_runs (on Success) and reruns (on NonSuccess). A passing Playwright testcase carrying a <flakyError> for a failed attempt was therefore emitted as a single SUCCESS run, so pass-on-retry flakes never surfaced. Emit an additional TestCaseRun (as a failure, with its own output and timing) for each parsed rerun/flaky attempt, in addition to the final attempt. The runs flow straight into the uploaded bin report, so no separate change is needed there. Fixes TRUNK-18796. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017d3M95mohe3LhktThN2rxz

trunk-io · 2026-07-13T22:53:22Z

😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details.

codecov-commenter · 2026-07-13T23:04:16Z

Codecov Report

❌ Patch coverage is 95.60976% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.00%. Comparing base (c052381) to head (7377709).

Files with missing lines	Patch %	Lines
context/src/junit/parser.rs	95.60%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1142      +/-   ##
==========================================
+ Coverage   82.52%   83.00%   +0.48%     
==========================================
  Files          70       70              
  Lines       15588    15788     +200     
==========================================
+ Hits        12864    13105     +241     
+ Misses       2724     2683      -41

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

trunk-staging-io · 2026-07-13T23:14:02Z

Failed Test	Failure Summary	Logs
`pending_quarantine_test should be quarantined when run with variant`	A test marked as pending was expected to fail but unexpectedly passed.	Logs ↗︎
`variant_quarantine_test should be quarantined when run with variant`	A test expected the sum of 2 + 2 to be 5, but it was actually 4, indicating a failing assertion.	Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2026-07-13T23:16:01Z

_{View Full Report ↗︎ ⋅ Docs}

upload_bundle and upload_bundle_using_bep hardcoded 500 test_case_runs, which only held because reruns were dropped. Now that each parsed rerun/flaky attempt is emitted as its own run, the total equals the test cases plus their (randomly generated) reruns. Compute the expected count from the bundled JUnit so the assertion is seed-independent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017d3M95mohe3LhktThN2rxz

dfrankland · 2026-07-14T05:49:54Z

+                    for rerun in &reruns {
+                        test_case_runs.push(test_case_run_for_rerun(&test_case_run, rerun));
+                    }


@claude instead of cloning TestReruns and setting the reruns variable, can we just iter over

let reruns = match &test_case.status { TestCaseStatus::Success { flaky_runs } => Some(flaky_runs.iter()), TestCaseStatus::NonSuccess { reruns, .. } => Some(reruns.iter()), TestCaseStatus::Skipped { .. } => None, }; if let Some(reruns) = reruns { for rerun in reruns { test_case_runs.push(test_case_run_for_rerun(&test_case_run, rerun)); } }

Done in 7377709 — now borrowing test_case.status when setting the run status so the attempts are iterated by reference (dropped the cloned Vec<TestRerun>), matching your snippet. Also switched the two rerun tests to a single assert_eq! over the whole test_case_runs vec per your other note.

Generated by Claude Code

dfrankland · 2026-07-14T05:52:13Z

        assert!(test_output.system_out.is_empty());
    }
+
+    #[test]


an aside: this file is too long, we should try to move these tests to the integration tests directory

Agreed the file has gotten long. I kept the test relocation out of this PR to keep the fix focused, but happy to move these (and the existing into_test_case_runs tests) into the integration tests dir as a follow-up.

Generated by Claude Code

dfrankland · 2026-07-14T05:53:28Z

+        assert_eq!(test_case_runs.len(), 2);
+
+        let rerun = &test_case_runs[0];
+        let final_run = &test_case_runs[1];
+
+        // The failed attempt comes first, the final passing attempt last.
+        assert_eq!(rerun.status, TestCaseRunStatus::Failure as i32);
+        assert_eq!(final_run.status, TestCaseRunStatus::Success as i32);
+
+        // Both attempts describe the same test.
+        assert_eq!(rerun.name, "should show error toast");
+        assert_eq!(final_run.name, "should show error toast");
+        assert_eq!(rerun.parent_name, final_run.parent_name);
+        assert_eq!(rerun.classname, final_run.classname);
+
+        let test_output = rerun
+            .test_output
+            .as_ref()
+            .expect("expected test output for flaky attempt");
+        assert_eq!(test_output.message, "Test timeout of 5000ms exceeded.");
+        assert_eq!(test_output.system_out, "attempt output");
+        assert!(final_run.test_output.is_none());


nit, optional: rather than making many small assertions, you should be able to assert_eq! over the entire test_case_runs vec (same goes for below)

dfrankland · 2026-07-14T05:54:00Z

We'll definitely want to let this bake and test internally prior to releasing it

…n tests Address review feedback: - Borrow test_case.status when setting the run status so the rerun/flaky attempts can be iterated by reference at emit time, dropping the cloned Vec<TestRerun>. - Assert over the entire test_case_runs vec in the rerun tests rather than field-by-field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017d3M95mohe3LhktThN2rxz

TylerJang27

Code looks good. I will enqueue after I test this myself manually. We may wish to expose an env var in the future that flattens test results in an upload to only their terminal status

TylerJang27 · 2026-07-14T13:18:39Z

Manually verified:

Prereqs or nothing changes for Playwright:

Playwright config with ["junit", { outputFile: commonTestResultsDir, includeRetries: true }]

File set up to intentionally flake:

import { test, expect } from "../fixtures";

test.describe("deliberately flaky", () => {
  test("fails twice, then passes on the third attempt", async ({
    page,
  }, testInfo) => {
    await page.goto("/");

    // testInfo.retry: 0 (initial), 1 (retry #1), 2 (retry #2). With retries: 2
    // this fails the first two attempts and passes on the third.
    expect(
      testInfo.retry,
      `attempt ${testInfo.retry + 1} — passes only on the 3rd`,
    ).toBe(2);
  });
});

Run playwright and generate the xml:

<testsuites id="" name="" tests="1" failures="0" skipped="0" errors="0" time="13.976637">
<testsuite name="deliberately-flaky.pwtest.ts" timestamp="2026-07-14T13:09:35.555Z" hostname="chromium" tests="1" failures="0" skipped="0" time="2.305" errors="0">
<testcase name="deliberately flaky › fails twice, then passes on the third attempt" classname="deliberately-flaky.pwtest.ts" time="0.666">
<flakyFailure message="attempt 1 — passes only on the 3rd" type="expect.toBe" time="0.816">
<stackTrace>
<![CDATA[Error: attempt 1 — passes only on the 3rd

expect(received).toBe(expected) // Object.is equality

Expected: 2
Received: 0
    at /home/tyler/repos/trunk2/ts/apps/e2e/__tests__/deliberately-flaky.pwtest.ts:14:7]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/video.webm]]

[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/error-context.md]]

[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/trace.zip]]
]]>
</system-out>
</flakyFailure>
<flakyFailure message="attempt 2 — passes only on the 3rd" type="expect.toBe" time="0.823">
<stackTrace>
<![CDATA[Error: attempt 2 — passes only on the 3rd

expect(received).toBe(expected) // Object.is equality

Expected: 2
Received: 1
    at /home/tyler/repos/trunk2/ts/apps/e2e/__tests__/deliberately-flaky.pwtest.ts:14:7]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/video.webm]]

[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/error-context.md]]

[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/trace.zip]]
]]>
</system-out>
</flakyFailure>
</testcase>
</testsuite>
</testsuites>

Run the analytics-cli from this branch, get an internal.bin with:

{
  "test_results": [
    {
      "test_case_runs": [
        {
          "id": "",
          "name": "deliberately flaky › fails twice, then passes on the third attempt",
          "classname": "deliberately-flaky.pwtest.ts",
          "file": "",
          "parent_name": "deliberately-flaky.pwtest.ts",
          "line": 0,
          "status": 2,
          "attempt_number": 0,
          "started_at": "2026-07-14T13:09:35.555Z",
          "finished_at": "2026-07-14T13:09:36.371Z",
          "status_output_message": "attempt 1 — passes only on the 3rd",
          "is_quarantined": false,
          "codeowners": [],
          "attempt_index": null,
          "line_number": null,
          "test_output": {
            "text": "",
            "message": "attempt 1 — passes only on the 3rd",
            "system_out": "[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/video.webm]]\n\n[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/error-context.md]]\n\n[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium/trace.zip]]",
            "system_err": ""
          },
          "test_runner_information": null
        },
        {
          "id": "",
          "name": "deliberately flaky › fails twice, then passes on the third attempt",
          "classname": "deliberately-flaky.pwtest.ts",
          "file": "",
          "parent_name": "deliberately-flaky.pwtest.ts",
          "line": 0,
          "status": 2,
          "attempt_number": 0,
          "started_at": "2026-07-14T13:09:35.555Z",
          "finished_at": "2026-07-14T13:09:36.378Z",
          "status_output_message": "attempt 2 — passes only on the 3rd",
          "is_quarantined": false,
          "codeowners": [],
          "attempt_index": null,
          "line_number": null,
          "test_output": {
            "text": "",
            "message": "attempt 2 — passes only on the 3rd",
            "system_out": "[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/video.webm]]\n\n[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/error-context.md]]\n\n[[ATTACHMENT|../apps/e2e/test-results/deliberately-flaky.pwtest.-eb968-passes-on-the-third-attempt-chromium-retry1/trace.zip]]",
            "system_err": ""
          },
          "test_runner_information": null
        },
        {
          "id": "",
          "name": "deliberately flaky › fails twice, then passes on the third attempt",
          "classname": "deliberately-flaky.pwtest.ts",
          "file": "",
          "parent_name": "deliberately-flaky.pwtest.ts",
          "line": 0,
          "status": 1,
          "attempt_number": 0,
          "started_at": "2026-07-14T13:09:35.555Z",
          "finished_at": "2026-07-14T13:09:36.221Z",
          "status_output_message": "",
          "is_quarantined": false,
          "codeowners": [],
          "attempt_index": null,
          "line_number": null,
          "test_output": null,
          "test_runner_information": null
        }
      ],
      "uploader_metadata": {
        "version": "",
        "origin": "",
        "upload_time": null,
        "variant": ""
      },
      "test_build_information": null
    }
  ],
  "uploader_metadata": {
    "version": "",
    "origin": "",
    "upload_time": null,
    "variant": ""
  }
}

Beforehand this just resulted in:

{
  "test_results": [
    {
      "test_case_runs": [
        {
          "id": "",
          "name": "deliberately flaky › fails twice, then passes on the third attempt",
          "classname": "deliberately-flaky.pwtest.ts",
          "file": "",
          "parent_name": "deliberately-flaky.pwtest.ts",
          "line": 0,
          "status": 1,
          "attempt_number": 0,
          "started_at": "2026-07-14T13:09:35.555Z",
          "finished_at": "2026-07-14T13:09:36.221Z",
          "status_output_message": "",
          "is_quarantined": false,
          "codeowners": [],
          "attempt_index": null,
          "line_number": null,
          "test_output": null,
          "test_runner_information": null
        }
      ],
      "uploader_metadata": {
        "version": "",
        "origin": "",
        "upload_time": null,
        "variant": ""
      },
      "test_build_information": null
    }
  ],
  "uploader_metadata": {
    "version": "",
    "origin": "",
    "upload_time": null,
    "variant": ""
  }
}

#1143) * fix(context): keep reruns emitted before the closing <failure>/<error> Playwright's junit reporter (with includeRetries) emits a test that failed every attempt as a single <testcase> whose <rerunFailure>/<rerunError> attempts appear BEFORE the closing <failure>/<error>. The parser starts each testcase as Success, so those reruns are parsed into flaky_runs while the status is still Success. set_test_case_status then replaced the whole status enum with a fresh NonSuccess (empty reruns), discarding every earlier attempt — so a fail-all-retries test collapsed to a single run and pass-on-retry never surfaced for the failing case. Carry the accumulated reruns onto the terminal status (mem::take the flaky_runs and add_reruns them to the new failure/error) so each attempt surfaces as its own run. The pass-on-retry (flaky) path was unaffected because it has no closing <failure> to clobber the runs. Adds a regression test using Playwright's real element order (system-out -> rerunFailure... -> failure); the existing failing-rerun test only passed because it ordered <failure> before <rerunFailure>. * test(context): make the fail-all-retries regression test faithful to Playwright The original #1142 test ordered <failure> before <rerunFailure>, and real Playwright output is the reverse. Rework the regression test to mirror the actual reporter output for a test that failed every attempt: reruns emitted BEFORE the terminal element, a mix of <rerunFailure> (assertion) and <rerunError> (timeout), and a terminal <error> (attempt 0 timed out) rather than <failure>. Asserts all four attempts surface as their own runs with each attempt's message preserved. Verified it fails without the fix. --------- Co-authored-by: Clippy (Triage) <clippy@trunk.io>

dfrankland approved these changes Jul 14, 2026

View reviewed changes

TylerJang27 approved these changes Jul 14, 2026

View reviewed changes

trunk-io Bot merged commit 7704b9d into main Jul 14, 2026
27 checks passed

mmatheson mentioned this pull request Jul 15, 2026

fix(context): keep reruns emitted before the closing failure/error tag #1143

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Surface individual rerun attempts as separate test runs#1142

Surface individual rerun attempts as separate test runs#1142
trunk-io[bot] merged 3 commits into
mainfrom
claude/playwright-reruns-flaky-fix-uor4fb

EliSchleifer commented Jul 13, 2026

Uh oh!

trunk-io Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 13, 2026 •

edited

Loading

Uh oh!

trunk-staging-io Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

dfrankland Jul 14, 2026

Uh oh!

EliSchleifer Jul 14, 2026

Uh oh!

dfrankland Jul 14, 2026

Uh oh!

EliSchleifer Jul 14, 2026

Uh oh!

dfrankland Jul 14, 2026

Uh oh!

dfrankland commented Jul 14, 2026

Uh oh!

TylerJang27 left a comment

Uh oh!

TylerJang27 commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

EliSchleifer commented Jul 13, 2026

Summary

Key Changes

Implementation Details

Uh oh!

trunk-io Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

trunk-staging-io Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfrankland Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

EliSchleifer Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

dfrankland Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

EliSchleifer Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

dfrankland Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

dfrankland commented Jul 14, 2026

Uh oh!

TylerJang27 left a comment

Choose a reason for hiding this comment

Uh oh!

TylerJang27 commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

trunk-io Bot commented Jul 13, 2026 •

edited

Loading

codecov-commenter commented Jul 13, 2026 •

edited

Loading

trunk-staging-io Bot commented Jul 13, 2026 •

edited

Loading

trunk-io Bot commented Jul 13, 2026 •

edited

Loading