Gemma4 ov int4 and stateful fixes#260
Draft
cavusmustafa wants to merge 6 commits into
Draft
Conversation
cavusmustafa
commented
Jul 18, 2026
cavusmustafa
commented
Collaborator
- enables gemma4 dense stateful support
- optional int4 downscale for requant: sacrifice accuracy for performance (env to enable - GGML_OPENVINO_INT4_REQUANT)
Adds support for the Gemma-4 26B-A4B hybrid-FFN MoE model on the OpenVINO backend (dense shared FFN + 128-expert top-8 MoE per layer, interleaved SWA/global attention, QK-norm, sandwich norms, logit soft-cap). Backend changes (all under ggml/src/ggml-openvino/): - 3D quantized expert weights: rank-2 GatherCompressed-matchable dequant (Q4_K gate/up, Q5_1 down), f16 zero-point (zp = -bias/scale) so the fusable Subtract form stays algebraically w*s + min without OOM; extracted in-place into the backend buffer (use_bias sizing). - Per-expert VIEW slicing on the non-static path + unique VIEW output names so the 8 same-named expert views no longer collide in the tensor_map. - MoE token dim kept dynamic through SUM_ROWS/DIV/CLAMP routing-norm ops and the per-expert scale, so all RoPE concats stay dynamic (fixes decode Broadcast mismatch and the GPU in-place-concat KV-cache corruption). - GET_ROWS batched-gather index broadcast tied to the data batch dim. - Full-MoE path: keep the whole MoE (routing gather/softmax/normalization + expert matmuls) on one OV submodel instead of fragmenting at every MoE node. Auto-enabled for MoE models on the dynamic-shape devices (CPU/GPU), latched on MUL_MAT_ID at placement; NPU keeps the fragmented static path. The un-fragmented graph is numerically correct on both CPU and GPU and avoids the cross-submodel index corruption the fragmented path hit. - op gating: exclude fused TOPK_MOE (uses ARGSORT routing), i64 CONCAT, borderline q4_1/q5_1 n=256 GET_ROWS, and oversized non-expert MUL_MAT_ID. - Guard is_kvcache() against a null view_src on SCALE ops (gemma4 inp_scaled), which otherwise segfaults compute_llm_params. Verified against ravi9/dev_backend_openvino: gemma4 26B MoE CPU greedy "Paris is the capital of France"; gemma4 E2B on CPU/GPU; dense Llama-3.2-1B on CPU+GPU byte-identical to baseline; test-backend-ops OPENVINO CPU 2622/2622 and GPU 2576/2576.
…-matmul placement) The GPU full-MoE large-tmp exemption in mul_mat_id_requires_large_tmp used op names (ffn_moe_gate_up/ffn_moe_down or an empty reserve-pass name) to keep the expert matmuls on OpenVINO. At ubatch>=32 the ggml scheduler's reserve/measurement pass queries the expert down-proj MUL_MAT_ID under an auto-assigned name 'node_<N>' (ggml_graph_add_node), which matched neither, so the worst-case ~2GB temporary exceeded the 1 GiB cap and the op was placed on ggml-CPU. It then read OpenVINO-produced routing ids across the OV<->CPU split boundary -> garbage indexing -> hard segfault / 'i02 >= 0 && i02 < n_as' assert during prefill (llama-bench -ub 32/64). Fix: exempt structurally instead of by name. A genuine MoE-model expert matmul routes over a 3D quantized expert-weight tensor (as->ne[2] > 1 && quantized) whose 'as' lives in a WEIGHTS buffer; the scheduler's earliest reserve query runs before weights are bound and reports an ANY buffer under an empty name, so treat that empty-name case as an expert matmul too. test-backend-ops MUL_MAT_ID/_FUSION cases use named, non-weights 'as' tensors, so they still hit the cap and stay on CPU as before. Verified: 26B MoE GPU llama-bench no longer crashes at -ub 32 (pp64 1.95) / -ub 64; test-backend-ops OPENVINO GPU MUL_MAT_ID 532/532 and MUL_MAT_ID_FUSION 9/9; default build compiles.
Gemma-4 uses different attention head dimensions per layer type (sliding_attention head_dim=256, full_attention global_head_dim=512), so the single scalar model_params.head_size cannot describe every layer. The stateful KV-cache reshape applied that one value to all layers, corrupting the SWA layers and failing SDPA shape inference. - ggml-decoder.cpp: derive the KV head size from each tensor's own combined dim (n_heads_kv * head_size) instead of the global scalar. - permute.cpp: the stateful path carries hidden-state tensors in a rank-3 layout; drop the leading batch axis from the rank-4 perm when the input is rank-3 (Gemma-4's per-layer-embedding permute), so the Transpose order matches the input rank. Stateful execution (GGML_OPENVINO_STATEFUL_EXECUTION) now runs and produces correct output for Gemma-4 on CPU and GPU.
Decode on the OpenVINO GPU backend is weight-bandwidth bound. By default the backend faithfully preserves the GGUF precision, which keeps some weights at 8-bit where the native OpenVINO export uses int4. This optional env flag re-quantizes those weights to symmetric int4 (off by default; small accuracy reduction): - MoE down-projection experts (Q5_1/Q8_0 -> u8) to int4 group-64. group-64 is required for the GatherMatmul fusion: it needs (m*k)/group divisible by the per-expert output rows N; the down expert has k=704 (704/64=11; 704/128 would break the fold). - dense attention/FFN weights (Q6_K/Q5_K -> per-channel int8) to int4 group-128. The int4 dequant chain still folds to GatherMatmulCompressed, so the experts stay compressed. Measured on Gemma-4 26B-A4B (Arc B390): prefill +13%, decode +18%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.