feat(common-json): extract pluggable JsonTokenizer SPI

[jfallows]

Description

Promotes JsonTokenizer (common-json) from a single hard-coded internal class to an exported, pluggable SPI:

• JsonTokenizer is now an exported interface with a DirectBufferEx-based scan entry point (wrap(buffer, offset, limit) / wrap(buffer, offset, limit, boolean last) + parameterless advance()) instead of the previous InputStream-only advance(InputStream). window(int) remains a separate method, decoupled from wrap().
• New JsonTokenizerFactory SPI (ServiceLoader-based) with priority()/available() capability-probe selection and a static instantiate() loader.
• The former concrete internal.JsonTokenizer is renamed to JsonTokenizerImpl, registered via a new JsonTokenizerImplFactory and META-INF/services/module-info uses/provides.
• JsonParserImpl obtains its tokenizer through JsonTokenizerFactory.instantiate() and feeds it its own wrapped DirectBufferEx directly, rather than an InputStream. The existing InputStream-constructor compatibility path (including external re-wraps of a DirectBufferInputStreamEx-backed input between hasNext() calls) is preserved unchanged via a rewrapRawInputIfNeeded() helper — no eager reads, no checked-exception surface change.
• New JsonStringView (extends CharSequence, adds getSegment(): DirectBufferEx) is now the declared return type of JsonTokenizer.stringView(), and JsonGeneratorImpl opportunistically splices its bytes directly when writing — see below.

No external call sites change: neither model-json nor binding-mcp touch JsonTokenizer directly, both go through JsonParserEx/JsonPipeline.

Motivation and context

Decouples JsonParserImpl from a single hard-coded tokenizer implementation, allowing alternate JsonTokenizer providers to be plugged in via ServiceLoader and selected by priority with a capability probe (available()), while preserving default behavior with no available alternate provider.

Fixes # (none — see zilla-plus#861 for the motivating downstream use case)

JsonStringView: tokenizer-agnostic groundwork for zero-copy value splicing

JsonTokenizer.stringView() now declares JsonStringView (not plain CharSequence) as its return type. JsonStringView adds one method beyond CharSequence: getSegment(): DirectBufferEx, a nullable hint — the contiguous run of original source bytes a value's characters were decoded from, when such a run exists and corresponds 1:1 to the decoded content, or null when decoding transformed the bytes (an escape sequence, a folded multi-byte UTF-8 sequence) and there is no single source byte range left to point at. When non-null, every byte in the segment is guaranteed plain ASCII and never requires escaping — a hard invariant documented on the interface, since it's what lets a consumer splice the bytes directly with no re-encoding pass, and lets the segment's byte length be relied on as equal to the view's char length.

JsonTokenizerImpl always decodes into scratch (a StringBuilder), so it never has a byte-backed alternative to offer — its stringView() wraps whatever CharSequence it already returns in a small reused DecodedStringView flyweight whose getSegment() unconditionally returns null. This is a correct, honest answer for this tokenizer, not a placeholder.

JsonParserImpl needed no changes. getStringView()'s existing branch for verbatim/segment values already returns tokenizer.stringView() directly, unwrapped, so a tokenizer that does produce a real segment-backed JsonStringView will have that type flow through to callers automatically. The other branch (decoded char-scalars, tracking bounded/resumable output-write progress via stringViewOffset) is unaffected: its own StringView wrapper deliberately stays plain CharSequence, since decoded content has no segment to offer regardless of what wraps it.

JsonGeneratorImpl now consumes this, in both write(CharSequence) (unbounded) and write(CharSequence, Completion) (bounded): an instanceof JsonStringView check with a non-null getSegment() splices the segment's bytes directly via a bulk buffer copy instead of the codepoint-by-codepoint escape path — safe unconditionally per the ASCII/no-escaping invariant above. The bounded path only takes this shortcut when the whole segment fits the current output window; a segment too large for one call declines and falls through unchanged to the existing codepoint path, which already knows how to bound and resume a partial write correctly. This deliberately does not attempt to fast-path a multi-call resumed write of an oversized segment-backed value — that would require reasoning about how a future JsonTokenizer producer re-exposes "the remainder" of a segment across calls, which is that producer's contract to keep, not this generator's to assume; the fast path here only ever claims "this exact instance, on this exact call, either fits or it doesn't."

This is deliberately narrow, tokenizer-agnostic groundwork, not a full feature: no production JsonTokenizer implementation in this repo produces a real segment-backed JsonStringView yet (JsonGeneratorStringViewTest uses a minimal test fixture to verify the generator side of the contract independently). It establishes the shared vocabulary a future vectorized JsonTokenizer provider (zilla-plus) would need to plug into and get this benefit automatically, with no further changes on this side. A larger, related migration of common-json's exported buffer surface from DirectBufferEx to MemorySegment was considered and explicitly deferred until MemorySegment/asSlice() gets Project Valhalla's value-class optimization, so it can be done together with proper performance validation rather than piecemeal; this JsonStringView addition does not depend on or block that.

wrap() takes DirectBufferEx instead of MemorySegment; terminal() folds into wrap()

The scan entry point originally introduced in this PR took a raw MemorySegment plus a separate terminal(boolean) mutator. Both are replaced:

• wrap(DirectBufferEx buffer, int offset, int limit) / wrap(DirectBufferEx buffer, int offset, int limit, boolean last) — mirroring JsonParserEx's own two-overload shape one layer up, instead of a bespoke tokenizer-only contract.
• The standalone terminal(boolean) mutator is removed; last is now a parameter of the four-argument wrap() overload, set atomically with the window rebind. The three-argument overload only rebinds the window and leaves terminal state untouched — this matches the prior behavior exactly, since JsonParserImpl's own three-argument wrap() never called terminal() either.

This was previously forcing every caller of JsonTokenizer.wrap() to first unwrap a DirectBufferEx down to its raw backing MemorySegment and manually add wrapAdjustment() before calling in — JsonParserImpl had two such call sites doing that arithmetic redundantly. JsonTokenizerImpl now reads bytes via DirectBufferEx.getByte(index), which already applies wrapAdjustment() internally, so both call sites simply pass their DirectBufferEx straight through. New JsonTokenizerWrapTest locks in the two-overload contract directly against JsonTokenizerImpl: a four-argument wrap(..., true) completes a bare trailing number at EOF, a four-argument wrap(..., false) correctly starves instead, and a subsequent three-argument wrap() call leaves a previously-set terminal state unchanged across a reset().

Fused integral number decode: numberIntegral()/numberInLongRange()/numberLongValue()

Adds three methods to JsonTokenizer, letting JsonParserImpl.getInt()/getLong() skip a second CharSequence-based pass through Integer.parseInt/Long.parseLong for the common case of a plain integral value. JsonTokenizerImpl.validateNumberChar — the existing RFC 8259 number-grammar state machine, already advanced one char at a time — now also accumulates sign and magnitude for the leading integer-part digits alongside its existing state transitions, so the value is available for free once the number completes.

numberMagnitude only accumulates up to 18 digits (LONG_SAFE_DIGITS): at or under that many digits a signed long cannot overflow regardless of sign, so no per-digit overflow check is needed. A longer integer (rare — both Long.MIN_VALUE and Long.MAX_VALUE are 19 digits) reports numberInLongRange() false, and getInt()/getLong() fall back to the original CharSequence-based path unchanged — the fast path only ever has to be correct for the case it claims to handle; every other case (non-integral values, out-of-range values, all existing NumberFormatException-throwing misuse) keeps its exact prior behavior via that fallback. isIntegralNumber() similarly reads numberState directly (ZERO/INT are the only accepting states never reached via a DOT/EXP transition) instead of re-scanning the CharSequence for ./e/E.

New JsonNumberDecodeBM isolates getLong() from the rest of the parse over a 2000-element number array. Before/after (stashing the fused implementation and re-running the identical benchmark): 5,288 → 5,622 ops/s (+6.3%), gc.alloc.rate.norm unchanged at ~1.3 B/op (Long.parseLong(CharSequence, ...) was already non-allocating; the win is skipping its virtual charAt() calls and redundant validity re-checking of digits the tokenizer's own grammar already validated). New JsonTokenizerNumberValueTest pins the three methods directly at the tokenizer level, including the 18-vs-19-digit fast-path boundary; two new JsonParserTest cases pin that same boundary and -0 handling at the JsonParserImpl level.

This is the interface-level piece zilla-plus#902 called out as needing a cross-repo design before attempting (its own vectorized digit-extent scan, scanDigitRun, currently only counts digits and never touches their values) — implemented here first since it's SIMD-agnostic; the SIMD provider adopting it is separate follow-up work in that repo.

Test plan

• ./mvnw install -pl runtime/common-json: checkstyle, license, and full test suite pass — 4851 tests, 0 failures, 0 errors.
• New tests: JsonTokenizerFactoryTest (priority ordering, availability filtering, throws-when-none-available, real ServiceLoader discovery of the built-in factory), updated JsonTokenizerPathTest (migrated to the DirectBufferEx-based entry point), JsonTokenizerStringViewTest (key/escaped-string/number stringView() calls all correctly report a null getSegment() for this tokenizer), JsonGeneratorStringViewTest (the generator's fast path fires and produces correct output when a segment fits, both unbounded and bounded; correctly falls back when the segment is null or too large for the current output window, producing byte-identical output to the plain-String path in that case), JsonTokenizerWrapTest (the wrap()/last contract described above), and JsonTokenizerNumberValueTest/two new JsonParserTest cases (the fused number-decode contract described above).
• JMH smoke pass on JsonPipelineBM after the DirectBufferEx.getByte() swap shows no allocation regression (near-zero gc.alloc.rate.norm on the pipeline paths, unchanged ~968 B/op on the schema-validation paths that were already allocating for unrelated reasons). New JsonNumberDecodeBM measures the number-decode fusion specifically (see above).
• Downstream consumers model-json and binding-mcp built and their test suites run against the changed common-json with no regressions and no call-site changes required.
• Moditect module packaging (./mvnw clean install -pl runtime/common-json) verified clean.

---

Generated by Claude Code