Skip to content

feat: add MacOS support#81

Open
not-matthias wants to merge 6 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos
Open

feat: add MacOS support#81
not-matthias wants to merge 6 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos

Conversation

@not-matthias

Copy link
Copy Markdown
Member

No description provided.

@codspeed-hq

codspeed-hq Bot commented Jun 2, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 54.04%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
❌ 3 regressed benchmarks
✅ 213 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Memory test_recursive_cached_fibo_30 1 B 29,968 B -100%
Simulation test_recursive_fibo_10 139 µs 276.3 µs -49.67%
WallTime switch 2 300 ns 336 ns -10.71%
Memory wait 1ms 16 B 10 B +60%
Simulation one 589.9 µs 402.7 µs +46.5%
WallTime test sync baz 10 384 ns 324 ns +18.52%
WallTime switch 1 324 ns 276 ns +17.39%
WallTime test sync baz 10 360 ns 312 ns +15.38%
Simulation wait 500ms 12.2 ms 10.7 ms +14%
WallTime iterative fibo 15 312 ns 276 ns +13.04%
WallTime iterative fibo 20 360 ns 324 ns +11.11%
WallTime test_iterative_fibo_10 396 ns 360 ns +10%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos (f2b5162) with main (4dae798)

Open in CodSpeed

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 2 times, most recently from f0b1349 to 3982c34 Compare June 2, 2026 10:36
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 32c3b31 to 1d75a61 Compare June 3, 2026 13:23
@not-matthias not-matthias marked this pull request as ready for review June 3, 2026 15:22
@greptile-apps

greptile-apps Bot commented Jun 3, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds macOS walltime support to the CodSpeed Node.js plugins by replacing the __codspeed_root_frame__ function-wrapping approach with timestamp-based marker pairs (currentTimestamp + addMarker), and introduces a dedicated codspeed-walltime-macos CI job that runs a macOS-only vitest benchmark.

  • Measurement strategy: packages/vitest-plugin/src/walltime/index.ts now captures currentTimestamp() before/after originalRun and emits MARKER_TYPE_BENCHMARK_START/END markers per task; packages/tinybench-plugin/src/shared.ts adds an equivalent sendRunMarkers that emits a single pair covering the entire batch run (walltime mode only).
  • CI restructuring: The old walltime-macos-test job is removed from ci.yml and replaced by codspeed-walltime-macos in codspeed.yml, which targets only macos.bench.ts to avoid duplicate uploads of the Linux benchmarks.
  • Submodule bump: packages/core/src/native_core/instruments/hooks is updated to expose the new currentTimestamp and addMarker native APIs.

Confidence Score: 4/5

Safe to merge once the staging upload URL is resolved; the measurement logic changes are well-structured.

The macOS CI job hardcodes the staging upload URL, meaning every macOS benchmark upload silently goes to staging rather than production. No macOS walltime data will ever appear in the production dashboard until this is corrected. All other changes look correct.

.github/workflows/codspeed.yml — the upload-url field in codspeed-walltime-macos points at the staging API.

Important Files Changed

Filename Overview
.github/workflows/codspeed.yml Adds codspeed-walltime-macos job, but hardcodes the staging upload URL (api.staging.preview.codspeed.io) which will send all macOS benchmark data to the staging environment rather than production.
.github/workflows/ci.yml Removes the old walltime-macos-test CI job that was replaced by codspeed-walltime-macos in codspeed.yml.
packages/vitest-plugin/src/walltime/index.ts Replaces the __codspeed_root_frame__ function wrapper approach with timestamp-based markers (currentTimestamp + addMarker) per task, aligning the vitest plugin with the new marker-based walltime measurement strategy.
packages/tinybench-plugin/src/shared.ts Adds sendRunMarkers that emits a single BENCHMARK_START/BENCHMARK_END marker pair (in walltime mode only) covering the entire batch run, gated by getInstrumentMode() !== "walltime".
packages/vitest-plugin/benches/macos.bench.ts New macOS-only benchmark file that skips on non-Darwin platforms; run exclusively by the codspeed-walltime-macos CI job to avoid duplicate uploads.
packages/tinybench-plugin/tests/index.integ.test.ts Adds mocks for the two new InstrumentHooks methods (currentTimestamp, addMarker) so the integration test compiles cleanly with the updated shared.ts.
packages/core/src/native_core/instruments/hooks Submodule pointer bumped to b9ddb5b to pick up the new native currentTimestamp and addMarker APIs required by this PR.

Sequence Diagram

sequenceDiagram
    participant VW as WalltimeRunner
    participant TB as WalltimeBenchRunner
    participant IH as InstrumentHooks
    participant CS as CodSpeed API

    Note over VW: Per-task flow (vitest)
    VW->>IH: startBenchmark()
    VW->>IH: currentTimestamp() runStart
    VW->>IH: originalRun.call(this)
    VW->>IH: currentTimestamp() runEnd
    VW->>IH: stopBenchmark()
    VW->>IH: addMarker(BENCHMARK_START, runStart)
    VW->>IH: addMarker(BENCHMARK_END, runEnd)
    VW->>IH: setExecutedBenchmark(pid, uri)

    Note over TB: Batch flow (tinybench)
    TB->>IH: currentTimestamp() batchStart
    loop each task
        TB->>IH: startBenchmark()
        TB->>IH: task.run()
        TB->>IH: stopBenchmark()
        TB->>IH: setExecutedBenchmark(pid, uri)
    end
    TB->>IH: currentTimestamp() batchEnd
    TB->>IH: addMarker(BENCHMARK_START, batchStart)
    TB->>IH: addMarker(BENCHMARK_END, batchEnd)
    IH-->>CS: upload results (walltime)
Loading

Reviews (13): Last reviewed commit: "chore: run on staging" | Re-trigger Greptile

Comment thread packages/vitest-plugin/src/walltime/index.ts Outdated
Comment thread packages/vitest-plugin/benches/macos.bench.ts Outdated
Comment thread .github/workflows/codspeed.yml Outdated
Comment thread .github/workflows/codspeed.yml Outdated
working-directory: packages/vitest-plugin
run: pnpm turbo run bench --env-mode=loose --filter=@codspeed/vitest-plugin
mode: walltime
runner-version: branch:sip-resign-exec-redirect

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Mutable branch reference for runner-version

runner-version: branch:sip-resign-exec-redirect points to a live branch that can be force-pushed or rebased at any time. If the branch tip changes between two benchmark runs, the profiling behaviour changes silently, making measurement results non-comparable across commits. Consider pinning to a specific commit SHA or tag once the feature branch is stable, e.g. runner-version: sha:<commit>.

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 4 times, most recently from 884b240 to 762c0d8 Compare June 4, 2026 16:04
@codspeed-hq

codspeed-hq Bot commented Jun 4, 2026

Copy link
Copy Markdown

Unable to generate the flame graphs

The performance report has correctly been generated, but there was an internal error while generating the flame graphs for this run. We're working on fixing the issue. Feel free to contact us on Discord or at support@codspeed.io if the issue persists.

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 762c0d8 to 93cf800 Compare June 4, 2026 16:41

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm

Comment thread packages/vitest-plugin/src/walltime/index.ts Outdated
Comment thread .github/workflows/codspeed.yml Outdated
# twice for one commit is not supported.
run: pnpm turbo run bench --env-mode=loose --filter=@codspeed/vitest-plugin -- macos
mode: walltime
runner-version: branch:sip-resign-exec-redirect

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont forget to drop this

Add benches/macos.bench.ts guarded with describe.skipIf(!darwin) so it
only runs on macOS, and wire the walltime-macos-test CI job to execute it
via a direct `node vitest.mjs bench --run macos` invocation.
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 93cf800 to e08c11f Compare June 10, 2026 10:19
run: pnpm turbo run bench --env-mode=loose --filter=@codspeed/vitest-plugin -- macos
mode: walltime
runner-version: branch:main
upload-url: https://api.staging.preview.codspeed.io/upload

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Staging upload URL will suppress macOS results in production

upload-url: https://api.staging.preview.codspeed.io/upload routes every upload from this job to the staging environment. macOS benchmark results will never appear in the production CodSpeed dashboard for any commit — the data lands in staging silently. If the intent was to test the macOS runner before going to production, this URL should be removed (or replaced with the production endpoint) once validation is complete.

@avalanche-staging

Copy link
Copy Markdown

Congrats! CodSpeed is installed 🎉

🆕 1 new benchmark was detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Detected benchmark


Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants