fix(rspec): report correct status for pending/skipped examples by dfrankland · Pull Request #1136 · trunk-io/analytics-cli

dfrankland · 2026-07-06T22:24:09Z

Problem

The RSpec plugin didn't handle pending examples correctly, because RSpec expresses a pending example's outcome as the inverse of its body:

pending example	RSpec `status`	build	plugin reported (before)
body still fails (expectation met)	`:pending`	green	skipped ❌
body now passes (fixed → `PendingExampleFixedError`)	`:failed`	red	failure (unhelpful error)

Reporting the raw status meant a still-broken pending test was recorded as skipped — never counted as failing, so it could never be quarantined — while a fixed one was recorded as a failure. Genuine skip/xit (which RSpec also reports as :pending) couldn't be told apart from a failing pending test.

Fix

Split the status decision into TrunkAnalyticsListener#status_and_exception, which reports the outcome RSpec actually decided (which also matches the build result):

skip / xit (never ran) → skipped — detected explicitly via metadata[:skip], up front, since RSpec reports skips as :pending too
pending, body still failing → success (the "should fail" expectation was met)
pending, body now passing → failure (PendingExampleFixedError; error pulled from example.exception)

RSpec's own build pass/fail behavior is left untouched (#set_exception) — a fixed pending example still fails the run; we only change what status/error is reported to Trunk.

Tests

Adds test/pending_status_spec.rb to the gem suite. It runs real pending/skip examples inside an RSpec::Core::Sandbox (so the fixed-pending's failure doesn't fail this suite) and asserts the mapping against genuine RSpec behavior.

Verification

Built the native extension and ran the full gem suite locally: 18 examples, 0 failures (13 existing + 5 new). Also confirmed end-to-end that the reported statuses serialize as skipped / success / failure as expected.

🤖 Generated with Claude Code

trunk-io · 2026-07-06T22:24:12Z

😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details.

trunk-staging-io · 2026-07-06T22:36:36Z

Failed Test	Failure Summary	Logs
`variant_quarantine_test should be quarantined when run with variant`	A test expected the sum of 2 + 2 to be 5, but it was 4, indicating a failing assertion.	Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2026-07-06T22:37:26Z

_{View Full Report ↗︎ ⋅ Docs}

codecov-commenter · 2026-07-06T22:39:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.77%. Comparing base (b416a25) to head (4cf6afa).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1136      +/-   ##
==========================================
+ Coverage   82.51%   82.77%   +0.26%     
==========================================
  Files          70       70              
  Lines       15588    15588              
==========================================
+ Hits        12862    12903      +41     
+ Misses       2726     2685      -41

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

RSpec expresses a `pending` example's outcome as the inverse of its body: a pending example whose body still fails is reported as :pending (build stays green — the "should fail" expectation was met), while one whose body now passes is reported as :failed via PendingExampleFixedError (build goes red). Split the status decision into TrunkAnalyticsListener#status_and_exception, which reports the outcome RSpec actually decided (which also matches the build result): - skip / xit (never ran) -> skipped (detected explicitly via metadata[:skip], since RSpec reports skips as :pending too) - pending, body still failing -> success (expectation met) - pending, body now passing -> failure (PendingExampleFixedError) RSpec's own build pass/fail behavior is left untouched. Adds a test in the gem suite that runs real pending/skip examples in an RSpec sandbox and asserts the mapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TylerJang27

One blocker on the smoke tests

TylerJang27 · 2026-07-07T13:25:38Z

                         started_at, finished_at, failure_message || '', backtrace || '', is_quarantined)
  end
+
+  # Determine the Trunk status to report for an example, plus the exception (if


Note that this could be a breaking change for users of TRUNK_LOCAL_UPLOAD_DIR, but I agree with your choices here

TylerJang27 · 2026-07-07T13:27:58Z

+      # Includes a pending example whose body unexpectedly passed: RSpec reports it
+      # as :failed with a PendingExampleFixedError (on example.exception) and breaks


TylerJang27 · 2026-07-07T13:30:46Z

Sanity check: this should rspec exit code 0, but we would see that is as nonzero nonquarantined failures.

Let's make sure this runs for a few cycles of smoke tests

You'll need to enumerate this file in .github/actions/test_ruby_gem_uploads/action.yaml, probs in run-regular-tests

I've configured the smoke test action to optionally run the unit tests with the compiled gem. The ruby.yaml won't run the unit tests, but the smoke_test.yaml and smoke_test_main.yaml will.

Exercise the gem's portable unit suite (rspec-trunk-flaky-tests/test) against the built, packaged gem in the periodic main-branch smoke workflow, without duplicating it on every PR. - test/ specs now `require 'rspec_trunk_flaky_tests'` (resolves to source under `rake test`, or the installed built gem in smoke) instead of require_relative-ing the source lib; Rakefile test task gains -Ilib. - The test_ruby_gem_uploads action gains a `run-gem-unit-suite` input (default false) that runs `rspec $GITHUB_WORKSPACE/rspec-trunk-flaky-tests/test` against the built gem with auto-upload disabled. - smoke_test_main.yml enables it (its checkout matches the built gem). PR callers (ruby.yml) leave it off, so `rake test` remains the sole PR run of these specs. smoke_test.yml (released gem) is intentionally left off to avoid source/release version skew. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

dfrankland force-pushed the rspec-pending-status-mapping branch from 298c6db to cb1482a Compare July 6, 2026 22:28

dfrankland requested a review from TylerJang27 July 6, 2026 22:30

dfrankland force-pushed the rspec-pending-status-mapping branch from cb1482a to e8b0c34 Compare July 6, 2026 22:49

TylerJang27 approved these changes Jul 7, 2026

View reviewed changes

dfrankland force-pushed the rspec-pending-status-mapping branch 2 times, most recently from 3d8fa1e to 46165da Compare July 7, 2026 17:54

dfrankland force-pushed the rspec-pending-status-mapping branch from 46165da to 4cf6afa Compare July 7, 2026 19:34

trunk-io Bot merged commit 3f8ab75 into main Jul 7, 2026
27 checks passed

TylerJang27 mentioned this pull request Jul 9, 2026

fix(rspec): quarantine fixed-pending examples instead of hard-failing #1141

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(rspec): report correct status for pending/skipped examples#1136

fix(rspec): report correct status for pending/skipped examples#1136
trunk-io[bot] merged 2 commits into
mainfrom
rspec-pending-status-mapping

dfrankland commented Jul 6, 2026

Uh oh!

trunk-io Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

trunk-staging-io Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 6, 2026 •

edited

Loading

Uh oh!

TylerJang27 left a comment

Uh oh!

TylerJang27 Jul 7, 2026

Uh oh!

TylerJang27 Jul 7, 2026

Uh oh!

TylerJang27 Jul 7, 2026

Uh oh!

dfrankland Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

		# Includes a pending example whose body unexpectedly passed: RSpec reports it
		# as :failed with a PendingExampleFixedError (on example.exception) and breaks

Uh oh!

Conversation

dfrankland commented Jul 6, 2026

Problem

Fix

Tests

Verification

Uh oh!

trunk-io Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-staging-io Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TylerJang27 left a comment

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

dfrankland Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

trunk-io Bot commented Jul 6, 2026 •

edited

Loading

trunk-staging-io Bot commented Jul 6, 2026 •

edited

Loading

trunk-io Bot commented Jul 6, 2026 •

edited

Loading

codecov-commenter commented Jul 6, 2026 •

edited

Loading