Orchestrator HTTP server spins at 100% CPU forever: node:http never reaps CLOSE_WAIT sockets (EPOLLRDHUP storm)

[chrisleekr]

Summary

The orchestrator's HTTP server burns exactly one full CPU core, continuously and indefinitely, once any upstream proxy half-closes a keep-alive connection. The process never recovers on its own.

Observed on chrisleekr/github-app:1.15.0-orchestrator (Bun 1.3.14) running behind ingress-nginx. The pod had burned 13.93 days of CPU over 13.95 days of wall-clock (99.88%) before it was restarted.

This is not a timer/poller problem, and not a proxy misconfiguration. It is a socket-lifecycle defect in Bun's node:http layer that this app is exposed to because src/app.ts never sets a server timeout.

Impact

• ~1 CPU core wasted per orchestrator pod, permanently, from ~20 minutes after startup.
• The spin is binary, not cumulative: a single stuck socket pegs the event-loop thread at 100%. Seven stuck sockets cost exactly the same as one. This is why CPU is flat rather than ramping, and why it is easy to mistake for "normal load".
• Restarting the pod is not a workaround. The pod was idle for only ~22 minutes of its 13.95-day life, i.e. the spin re-established itself within ~20 minutes of boot.

Evidence

strace on the main thread (PID 1), ~8 seconds:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 62.75    2.732801           5    517658           clock_gettime
 36.99    1.610774           6    258776           epoll_pwait2
  0.09    0.003796          36       104        13 futex
  0.07    0.003124          72        43           sched_yield
  0.02    0.000804          15        53           recvfrom
  0.00    0.000031          15         2           close
  0.00    0.000013           6         2           timerfd_settime

epoll_pwait2 is called ~32,000 times per second with zero errors. clock_gettime runs at exactly 2.0002x that rate (the event loop timing each iteration).

The raw calls show exactly what it is spinning on:

epoll_pwait2(4, [{events=EPOLLRDHUP, data=0x575dfe20f00},
                 {events=EPOLLRDHUP, data=0x575dfe20400},
                 {events=EPOLLRDHUP, data=0x575dfe22700},
                 {events=EPOLLRDHUP, data=0x575dfe22b00},
                 {events=EPOLLRDHUP, data=0x575dfe20900},
                 {events=EPOLLRDHUP, data=0x575dfe22e00},
                 {events=EPOLLRDHUP, data=0x575dfe22200}],
             1024, {tv_sec=0, tv_nsec=78481990}, [], 8) = 7

Every call returns the same 7 fds, all EPOLLRDHUP, with identical data pointers. Bun asks to wait ~78ms (tv_nsec visibly counts down toward its next timer deadline across successive calls) but epoll returns instantly, because those 7 fds are permanently ready.

Those 7 fds correspond one-for-one with 7 sockets stuck in CLOSE_WAIT:

State       Recv-Q  Local Address:Port          Peer Address:Port
CLOSE-WAIT  1       [::ffff:10.0.0.206]:3000    [::ffff:10.0.0.67]:48232
CLOSE-WAIT  1       [::ffff:10.0.0.206]:3000    [::ffff:10.0.0.67]:42786
... (7 total)

10.0.0.67 is the ingress-nginx controller. Recv-Q=1 is the undrained FIN.

Supporting measurements:

• Exactly 1.000 cores, flat for 24h (Prometheus rate(container_cpu_usage_seconds_total[5m])).
• nr_throttled 0, throttled_usec 0 against cpu.max = 200000 100000 (a 2-core limit) — the process could use 2 cores and is not being capped. The spin is genuine.
• user_usec 56.0% / system_usec 44.0% — the kernel half is the epoll_pwait2 storm.
• Only 2.9 voluntary context switches/sec while burning a full core: the thread essentially never sleeps.
• Per-thread accounting: thread 1 holds 120,353,370 of 120,387,653 ticks (99.97%). Every other thread is idle.

Timers are excluded: timerfd_settime was called 2 times in 8 seconds. proposal-poller runs at intervalMs=90000 and scheduler at intervalMs=300000. Neither can produce a 32kHz loop.

Mechanism

1. ingress-nginx is configured with upstream-keepalive-timeout: 5 — it closes idle upstream connections after 5s. This is normal, correct proxy behaviour.
2. Bun's node:http server uses a default idle timeout of 10s (uWS HTTP_IDLE_TIMEOUT_S, see Ability to set keepAliveTimeout in server created by "node:http" like in Nodejs. oven-sh/bun#12446). Because 10s > 5s, nginx always closes first.
3. The peer's FIN arrives. Our socket enters CLOSE_WAIT. Bun registered EPOLLRDHUP interest on it.
4. Bun never handles the EPOLLRDHUP: it does not close() the fd, nor epoll_ctl(EPOLL_CTL_DEL) it. (close fired 2x and epoll_ctl 6x across 8 seconds — nowhere near the 7 needed.)
5. epoll is level-triggered, so the fd stays permanently ready. epoll_pwait2 returns immediately, forever. One core, gone.

It is a race, not a rule — nginx closed thousands of connections over 14 days and only 7 stuck. So timeout tuning changes the odds, not the mechanism.

Suggested fix

src/app.ts:326 never sets a server timeout:

const server = http.createServer((req, res) => { ... });   // line 244
server.listen(config.port, () => { ... });                 // line 326

Per a Bun maintainer in oven-sh/bun#12446, the supported knob for node:http servers in Bun is server.setTimeout() (since v1.1.27) — not server.keepAliveTimeout, which Bun does not honour the same way as Node.

Options, roughly in order of preference:

1. Set server.setTimeout() below the proxy's idle timeout so we close first and never enter CLOSE_WAIT. Note this inverts the usual advice (for AWS ALB you want the server timeout longer than the LB's, to avoid a 502 race). nginx's proxy_next_upstream error retries connection-level failures by default, which makes the trade more acceptable here than it would be behind an ALB — but it is a real trade and should be verified under load.
2. Move the server to Bun.serve(), which supports idleTimeout natively and avoids the node:http shim entirely. Larger change; the app is already on Hono, which has a native Bun adapter.
3. Report upstream to Bun. I could not find an existing issue for EPOLLRDHUP/CLOSE_WAIT spin in node:http. The closest is oven-sh/bun#33699 (node:http leaking on client disconnect, confirmed on 1.3.14, closed 2026-07-07) — same subsystem, different symptom. Bun 1.3.14 is the latest release (2026-05-13), so there is no version to upgrade to.

Whatever the app does, the underlying Bun defect should be fixed upstream: a half-closed peer connection must not be able to peg an event loop forever.

Environment

Image	chrisleekr/github-app:1.15.0-orchestrator
Bun	1.3.14 (latest release, 2026-05-13)
Server	node:http via http.createServer (src/app.ts:244), Hono 4.12.25
Platform	Kubernetes 1.33.4, Linux x64, Debian 13 (trixie) base
Proxy	ingress-nginx, upstream-keepalive-timeout: 5, upstream-keepalive-time: 30s
Container	requests: cpu=500m/512Mi, limits: cpu=2/2Gi, Burstable, caps dropped ALL, uid 1000

Reproduction sketch

Not yet reduced to a standalone case. Expected shape:

1. http.createServer under Bun 1.3.14, listening, no setTimeout set.
2. A client that opens keep-alive connections and half-closes them after a shorter idle period than Bun's 10s default (mimicking upstream-keepalive-timeout: 5).
3. Drive traffic until a connection lands in CLOSE_WAIT (it is a race; may take a while).
4. Observe: ss -tan state close-wait non-empty, and CPU pinned at exactly 1.00 core with strace showing epoll_pwait2 returning EPOLLRDHUP in a tight loop.

A minimal repro is the main thing needed before filing upstream with Bun.

[chrisleekr]

Pre-implementation review: the suggested fix is disproven, and two premises are wrong

I lab-tested the proposal before implementing it. All measurements below are from oven/bun:1.3.14 (Linux aarch64, Docker), reproduced twice — once by a research agent (~400 sockets), once by me directly. Caveat up front: aarch64, loopback, synthetic peer; production is x64 behind real nginx over 14 days.

1. The "Bun defaults to a 10s idle timeout" premise is false for node:http

An idle keep-alive socket against http.createServer was never closed — survived >14s in my run, 150s in the agent's. Property readback shows keepAliveTimeout: 5000, headersTimeout: 60000, requestTimeout: 300000, and server.timeout: undefined — all cosmetic; setting them is a silent no-op (verified: none closed a socket in 14s).

The 10s figure is real but belongs to Bun.serve, not node:http.

This doesn't weaken the report — it strengthens step 2 of the mechanism. nginx doesn't close first because 10s > 5s; it closes first because we never close at all, ever. That part of the diagnosis is correct, for a different reason.

2. server.setTimeout() is quantized to a 4s wheel and silently kills in-flight requests

setTimeout(2000) -> socket actually closed at 4002ms
ST=1000/2000/3000 -> all ~4000ms   ST=4000 -> 4015ms   ST=4001 -> 8008ms

ticks = ceil(ST/4000) — a global sweep at uSockets granularity, not a per-socket timer. Requested values below 4000ms are effectively ignored, and a socket going idle mid-tick gets an effective timeout anywhere in (0, 4000] (measured as low as 497ms from a requested 2000ms).

Worse, and this is the disqualifier:

setTimeout(2000) + handler that takes 6s  ->  client got FIN at ~4s, NEVER saw a response
control (no setTimeout)                   ->  clean 200

The handler completed and wrote into a dead socket. Silent data loss. server.on('timeout') does not suppress it (fires with destroyed:false, socket dies 3ms later anyway) — Bun does not honour Node's suppression semantics. To beat nginx's 5s you need ST <= 4000, which means every response slower than ~4s is dropped. POST /api/scheduler/run (enumerates installations, fetches configs from GitHub) can exceed that.

So Option 1 as written is unsafe here.

3. Option 2's rationale doesn't hold either

the app is already on Hono, which has a native Bun adapter

It isn't. src/app.ts:244 is raw node:http with a hand-rolled request handler. hono / @hono/node-server appear only as overrides pinning transitive deps of @modelcontextprotocol/sdk; nothing in src/ imports them. Separately, Bun.serve({idleTimeout}) inherits the identical 4s wheel and the identical in-flight kill — it is not an escape from (2).

(FWIW the WS server at src/orchestrator/ws-server.ts:207 is already Bun.serve with idleTimeout: 120 on a separate port — so no HTTP-side timeout can affect the daemon fleet. That risk is not real.)

4. I could not reproduce the CLOSE_WAIT spin — but that does not refute the report

~400 sockets across 7 half-close variants (stray-byte-then-FIN, pipelined-then-FIN, FIN-before-response, FIN mid-idle, same-tick, handler never consuming EOF): CLOSE-WAIT=0 at every probe, CPU ~0%, and strace -c showed 3 epoll_pwait2 calls in 3s — the inverse of a storm. Bun echoed the FIN in <1ms every time.

This is a negative at n≈400 for an event that hit ~7 times in what you describe as thousands of closes over 14 days (order 0.01–0.1%). Expected hits at n=400: well under 1. So my negative is exactly what a correct diagnosis would also predict. The production strace/ss evidence stands on its own; I just can't yet reproduce it on demand, which is precisely what filing upstream needs.

5. What actually works: server.closeIdleConnections()

closeIdleConnections()  ->  idle keep-alive socket reaped, ~immediate
                        ->  concurrent 6s in-flight request STILL GOT ITS RESPONSE

Bun 1.3.14 honours it with correct Node semantics: closes only connections with no in-flight request. That's the precise, safe primitive setTimeout is not.

Recommended corrected scope

The honest framing: we cannot ship a proven fix for the spin from app code, because the spin is a Bun-internal race we can't yet trigger, and if Bun orphans the fd below its own JS layer, no app-level watchdog can even see it.

What we can do, ordered:

1. Fix the proven defect. The server has no idle timeout at all. Idle conns accumulate indefinitely, and server.close() on SIGTERM (src/app.ts:751) can hang on them until the 290s force-exit. Real, testable, worth fixing regardless of the spin.
2. Optionally remove the spin's precondition via a closeIdleConnections() sweep tuned below nginx's 5s. This makes us close first, so the CLOSE_WAIT never forms. Trade: it inverts the standard proxy/backend advice (backend idle timeout should normally exceed the proxy's, so the proxy always closes) and reintroduces a small close-race that proxy_next_upstream error would have to absorb. Benefit is unproven against a defect I can't reproduce.
3. Upstream + repro. A half-closed peer pegging an event loop forever is a Bun bug regardless of what we do here. The minimal repro is still the blocker for filing.

Asking for a scope decision before writing any code, rather than shipping setTimeout() on a rationale the runtime contradicts.

[chrisleekr]

Investigation result: not implementing. The suggested fix is disproven, and the stated mechanism doesn't survive contact with the evidence.

Three independent labs (~28k upstream closes, arm64 + amd64, real nginx) plus live production inspection. Summary first, evidence below.

Do not ship server.setTimeout(). It is unsafe here, for reasons independent of whether the diagnosis is right.

---

1. Live production check: the pod is not wedged right now

Current container github-app-6c479979d7-qx887 (10.0.0.206 — the same IP as the ss output in the report):

startedAt   2026-07-17T11:22:07Z    (105 min uptime at time of check)
CPU         22m / 4m / 4m           (three samples, 20s apart)
traffic     87 x http.webhook.received in 110 min  (real nginx upstream churn, not just probes)

Not idle-by-absence-of-traffic — 87 webhooks actually flowed through nginx and it still did not wedge. The previous container terminated reason=Error with no crash trace; its final logs are routine fleet.snapshot lines, consistent with the spin never breaking functionality.

2. The report's numbers are internally contradictory

This is the load-bearing problem, and it is what turned me around:

• Observed traffic is ~0.8 req/min → ~16k upstream closes over 14 days ("thousands" ✓).
• 7 stuck / 16k = ~0.04% per close.
• At 0.04%, the first wedge should appear ~2,270 closes in ≈ ~2 days.
• But 13.93 CPU-days / 13.95 wall-days = 99.88% implies the spin began ~20 min after boot and never stopped.

A per-close race cannot produce both. If the rate were high enough to wedge within 20 min (~6%), 14 days would have wedged ~960 sockets, not 7. So "each nginx idle-close is an independent ~0.04% chance to wedge" is not what happened. Something else is going on — either one burst event wedged all 7 at once, or the trigger isn't the idle-close path at all.

I tested the burst theory (below). It's clean too. So the mechanism in the report is currently unsupported, not merely unreproduced.

3. What three labs actually showed

Lab	Exposure	Result
1 — synthetic staggered	~400 sockets, 7 half-close variants	0 CLOSE_WAIT, 0% CPU; strace -c = 3 epoll_pwait2 calls in 3s
2 — real nginx, staggered	~19.7k closes / ~263k reqs, arm64 + amd64, conc 1→300, bodies 0/1K/64K, busy+slow handlers	max CW 139 transient, longest run 10s, always drained; idle CPU 0.004 cores
3 — mass simultaneous FIN	25 nginx pool-drops (-s reload/SIGQUIT) + 456 bursts / 7,883 FINs, N=2→64, incl. Bun.serve WS listener in the same process	nginx reload: max CW = 0. Synthetic: 62–64 CW in one wakeup, longest-lived 655ms, 0 residual

Detector is proven, not assumed: a python server that ignores FIN yields 20/20 detected CLOSE_WAIT in the same netns. The zeros are real zeros.

Two structural findings that make a persistent CLOSE_WAIT hard to reach at all:

• Bun force-closes on FIN even with allowHalfOpen: true — 25/25 half-closed sockets closed anyway. Bun's TCP layer does not want to sit in CLOSE_WAIT.
• Recv-Q=1 means zero unread bytes, not one. Verified instrumentally (1 unread byte + FIN shows Recv-Q:2). The FIN itself counts as 1. So the production sockets are clean half-closes with nothing pending — which refutes the "undrained byte wedges the parser" reading.

Notable near-miss: at CONC=8 the nginx warm pool settled at exactly 7 upstream conns. So "7 = pool size at the moment of one event" is a very plausible shape — the pool-drop just doesn't wedge Bun in the lab.

4. Two premises in the report are factually wrong

"Bun's node:http default idle timeout is 10s." It is never. An idle keep-alive socket survives indefinitely (>150s measured). keepAliveTimeout: 5000 / headersTimeout: 60000 / requestTimeout: 300000 read back as values but are cosmetic no-ops; server.timeout is undefined. The 10s figure belongs to Bun.serve, not node:http.

This strengthens step 2 of the mechanism (nginx does always close first) while invalidating its reasoning (not "because 10s > 5s" — because we never close at all).

"The app is already on Hono." It isn't. src/app.ts:244 is raw node:http with a hand-rolled handler; hono/@hono/node-server exist only as overrides pinning transitive deps of @modelcontextprotocol/sdk and are imported nowhere in src/. Option 2 also gains nothing: Bun.serve({idleTimeout}) inherits the identical 4s wheel and identical in-flight kill (below).

(Also: the WS server at src/orchestrator/ws-server.ts:207 is a separate Bun.serve on a separate port with idleTimeout: 120, so no HTTP-side timeout can affect the daemon fleet.)

5. Why server.setTimeout() must not ship

setTimeout(2000)  ->  socket actually closed at 4002ms
ST=1000/2000/3000 -> all ~4000ms      ST=4000 -> 4015ms      ST=4001 -> 8008ms

ticks = ceil(ST/4000) — a global uSockets-granularity sweep, not a per-socket timer. Sub-4s values are ignored; a socket idling mid-tick gets an effective timeout anywhere in (0, 4000] (measured 497ms from a requested 2000ms).

And the disqualifier:

setTimeout(2000) + 6s handler  ->  client got FIN at ~4s, NEVER saw a response
control (no setTimeout)        ->  clean 200

The handler completed and wrote into a dead socket. Silent data loss. server.on('timeout') does not suppress it — Bun ignores Node's suppression semantics. Beating nginx's 5s requires ST <= 4000, i.e. dropping every response slower than ~4s. scheduler.scan already takes 13.5s (repos_enumerated: 41), so this is not hypothetical.

6. The safe primitive, for when we have proof

closeIdleConnections()  ->  idle keep-alive socket reaped immediately
                        ->  concurrent 6s in-flight request STILL GOT ITS RESPONSE

Bun honours server.closeIdleConnections() with correct Node semantics. If we ever decide to mitigate, this is the tool — not setTimeout. It remains unproven against the actual defect, and using it to close before nginx would invert the standard "backend idle > proxy idle" rule and introduce a new close-race, so I'm not shipping it on speculation.

Recommendation

Rescope this issue from "apply this fix" to "capture the production signature." Three clean labs at ~28k closes says the lab is missing an environmental ingredient, not that we haven't found the right close pattern. Blind lab hunting has hit diminishing returns.

Highest-value next step, when the pod next wedges:

ss -tanei                    # the stuck sockets: timers, state, real Recv-Q
strace -c -p 1               # confirm the epoll_pwait2 rate
strace -p 1 2>&1 | head -50  # confirm same fds + EPOLLRDHUP
cat /proc/1/status           # voluntary_ctxt_switches

Since it isn't wedged right now, the practical move is instrumentation that catches it unattended — sample /proc/net/tcp for CLOSE_WAIT on :3000 plus event-loop utilization, and emit a structured Pino event when CLOSE_WAIT persists across samples. That's a small, safe, provable change and it converts this from "unreproducible" to "diagnosable on next occurrence". Happy to ship that as its own issue if you want it.

Worth stating plainly: the production evidence (strace, ss, 99.88% CPU over 14 days) is real and I'm not disputing that something burned a core. What I can't support is the specific causal story or the proposed fix.

[chrisleekr]

Spun out #265 to instrument and bound this rather than guess at a fix.

To be explicit about scope: #265 does not fix this issue and #264 stays open. It makes the defect self-documenting on next occurrence (structured event carrying the fd table, socket ages, and CPU over the window) and optionally self-heals by exiting non-zero so k8s restarts the pod, capping a permanent 1-core burn.

It also settles a secondary hypothesis: this container exited reason=Error with no crash trace. If the stuck fds accumulate rather than staying at 7, a 14-day soak ending in EMFILE would explain both the burn and the silent non-zero exit. A persisted CLOSE_WAIT count over time proves or kills that.

Once #265 has captured a real occurrence, this issue gets the ss -tanei + strace -c signature it needs — either to fix here or to file upstream with Bun.

[chrisleekr]

Reopening: this was auto-closed when PR #266 merged (a closing-keyword in the squashed commit text matched this issue). #264 is NOT fixed and stays open. PR #266 shipped the instrumentation from #265 (detection + optional self-heal), which does not address the underlying Bun defect. This issue remains the tracking home for the actual root-cause fix once the new watchdog captures a live occurrence (socket.close_wait.spin_suspected with the fd table + CPU evidence).