diff --git a/docs/en/inference_guide/assets/minimax-m2.5-w8a8/minimax-m2.5-w8a8-agg-llmisvc.yaml b/docs/en/inference_guide/assets/minimax-m2.5-w8a8/minimax-m2.5-w8a8-agg-llmisvc.yaml new file mode 100644 index 0000000..93f58e8 --- /dev/null +++ b/docs/en/inference_guide/assets/minimax-m2.5-w8a8/minimax-m2.5-w8a8-agg-llmisvc.yaml @@ -0,0 +1,459 @@ +# MiniMax-M2.5 (W8A8) — plain `agg` aggregation on Alauda AI's InferNex surface: +# KServe `LLMInferenceService` + InferNex-Bridge + hermes-router, DP2 × TP=8 across +# TWO nodes (16 cards). This is the EXACT shape deployed and benchmarked on +# 2 × 8 × Ascend 910B3 (64 GB/card). +# +# MiniMax-M2.5 is the `minimax_m2` architecture (`MiniMaxM2ForCausalLM`): a 256-expert +# MoE (8 experts/token, NO shared expert) with 62 layers of **full attention** (GQA +# 48/8 heads, head_dim 128 — NOT a hybrid-KV / sliding-window model) and a native MTP +# head, served here as W8A8 (`W8A8_DYNAMIC`, QuaRot int8 attn+expert, no KV quant; +# ~230 GB weights, 71 shards, 62 layers). 230B total / ~10B activated per token. At +# ~230 GB the weights fit a single 8 × 64 GB node, but this variant runs a single +# aggregated service spanning TWO 64 GB nodes as **DP2 × TP=8 + EP16** — apples-to-apples +# with the DeepSeek-V4-Flash 16-card topology, and DP2 roughly doubles prefill throughput +# on the prefill-heavy benchmark. ~230 GB / 16 ≈ 14.4 GB/card leaves ~40 GB/card for KV + +# activations. It is a SINGLE service with replicas=1: the two DP ranks are the leader +# (rank 0) and worker (rank 1) of one LeaderWorkerSet, NOT two independent replicas. +# +# >>> This is a PLAIN aggregation (`agg`): NO mooncake KV store and NO speculative +# >>> decoding. That is the DELIBERATE PRODUCTION CONFIG for this model, not a first cut: +# >>> • MiniMax-M2.5 is "fast" (only ~10B activated + full attention + W8A8), so BOTH +# >>> acceleration layers backfire and were measured out: +# >>> - eagle3 speculative decoding: net LOSS at batch (S1 conc32 decode 377 vs +# >>> baseline 631 tok/s) AND a real 27k-prefill agent request returns HTTP 500 +# >>> (baseline serves the same request with 0 error). The model's built-in MTP +# >>> head is unusable — the QuaRot weights ship no MTP tensors (only up to layer 61). +# >>> - mooncake KV store: net LOSS even with a fully-warmed store (~91% merged +# >>> prefix hit). MiniMax prefill is so fast that re-fetching KV over the network +# >>> is >= recomputing it locally, so TTFT rises and throughput is flat/down. The +# >>> opposite of DeepSeek, where slow prefill makes the store a clear win. +# >>> • What IS a net win — and all this config keeps — is the FULL_DECODE_ONLY decode +# >>> graph plus the engine's LOCAL prefix cache (~62.6% hit on the real agent trace). +# +# Model source: ModelScope `Eco-Tech/MiniMax-M2.5-w8a8-QuaRot` (71 shards, ~230 GB), +# also repackaged as an Alauda ModelCar in OCI Image Layout format (multi-arch amd64+arm64): +# http://package-minio.alauda.cn:9199/packages/aml-models-oci/v0.1.0/MiniMax-M2.5-w8a8-QuaRot.oci.tar +# The benchmark itself staged the weights on a **node-local PV/PVC per node** (this file), +# which is what the manifest below mounts. The OCI ModelCar is the cross-environment +# distribution path: import it into your own registry once, then point model.uri at +# oci://… — see the page and the ALT note near model.uri below. NOTE: this ModelCar is +# >200 GB, so an `oci://` storageInitializer pull needs extra ephemeral space handling +# (disable the storageInitializer and extract to tmpfs) — the node-local PV/PVC path +# below is the validated one. +# +# Router = hermes-router (EPP) with the `random` picker and cache-indexer OFF: a single +# aggregated endpoint needs no global KV index / KV-cache-aware routing — this matches the +# GLM-5.2 / Qwen3-32B aggregation paradigm (router stays random, no indexer). Here the +# mooncake store is ALSO disabled (see the config decision above). +# +# Engine params (from vLLM-Ascend's `MiniMax-M2.md` §5.1 A2 + the QuaRot test config, +# adapted to DP2 × TP=8): `--quantization ascend` + `--trust-remote-code` + +# `--enable-expert-parallel` + `--tool-call-parser minimax_m2` + `--reasoning-parser +# minimax_m2` (one parser covers both agent structured reasoning AND aiperf timing on +# v0.23.0) + `--compilation-config FULL_DECODE_ONLY` + `--async-scheduling` + +# `--gpu-memory-utilization 0.85` (official value; long ctx, do not max out) + +# `--max-num-batched-tokens 16384` + `--max-num-seqs 32` + `--max-model-len 196608`; +# env `VLLM_ASCEND_ENABLE_FLASHCOMM1=1`. +# ⚠️ Do NOT set `enable_fused_mc2`: the vendor guide marks it "A3-recommended, not +# applicable to A2", and 910B3 is an A2-class part. +# +# Image: quay.io/ascend/vllm-ascend:v0.23.0-openeuler (vLLM 0.23.0, the release-pinned +# `nightly-releases-v0.23.0-openeuler` build). Chosen over the older nightly-main-0709 +# snapshot because the v0.23.0 release branch carries vLLM #11505 — streaming tool-call +# arguments are held until the closing `` so no trailing-tag fragment is +# spliced into a command argument (validated over 3,123 tool executions, 0 tail pollution). +# The tradeoff: a small decode regression vs 0709 (decode throughput -3…11% / ITL +5…18%, +# growing with concurrency; TTFT flat or better). Match the tag's CANN version to your +# host NPU driver; both nodes must have the image locally. +# +# Ingress: benchmarked through the product **MaaS gateway** (OpenAI-compatible, API-key +# auth, Envoy AI Gateway v0.6.0 OpenAI→OpenAI passthrough) as well as the internal KServe +# ingress — see the page for both curl forms. +# +# Before applying: set the namespace `demo`, stage the ~230 GB weights on a node-local +# directory on BOTH nodes and pin the two hostPath PVs to those node hostnames (replace +# / and the weights path below), and make sure both nodes can pull the +# engine + hermes-router images (air-gapped nodes need them pre-loaded). +# ============================================================================= +--- +# Weights are staged on each node's LOCAL disk (~230 GB per node) and exposed to the +# engine as a `pvc://` mount via a hostPath PV+PVC pinned to that node — this is the +# validated path (no 230 GB storageInitializer pull). Leader reads the node-0 copy. +apiVersion: v1 +kind: PersistentVolume +metadata: + name: minimax-n0 + labels: { model: minimax-m25-w8a8, node: "node-0" } +spec: + capacity: { storage: 300Gi } + accessModes: [ReadOnlyMany] + persistentVolumeReclaimPolicy: Retain + storageClassName: "" + hostPath: { path: /home/minimax-m25-w8a8, type: Directory } # <— node-local weights dir + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - { key: kubernetes.io/hostname, operator: In, values: [""] } +--- +# Worker reads the node-1 copy (byte-identical to node-0). +apiVersion: v1 +kind: PersistentVolume +metadata: + name: minimax-n1 + labels: { model: minimax-m25-w8a8, node: "node-1" } +spec: + capacity: { storage: 300Gi } + accessModes: [ReadOnlyMany] + persistentVolumeReclaimPolicy: Retain + storageClassName: "" + hostPath: { path: /home/minimax-m25-w8a8, type: Directory } # <— node-local weights dir + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - { key: kubernetes.io/hostname, operator: In, values: [""] } +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: minimax-n0 + namespace: demo +spec: + accessModes: [ReadOnlyMany] + storageClassName: "" + volumeName: minimax-n0 + resources: { requests: { storage: 300Gi } } +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: minimax-n1 + namespace: demo +spec: + accessModes: [ReadOnlyMany] + storageClassName: "" + volumeName: minimax-n1 + resources: { requests: { storage: 300Gi } } +--- +apiVersion: serving.kserve.io/v1alpha2 +kind: LLMInferenceService +metadata: + name: minimax-m25-w8a8-agg + namespace: demo + labels: + infernex.io/runtime: "true" # triggers InferNex-Bridge reconcile + annotations: + # agg (baseline): mooncake is disabled together with cache-indexer/pd-orchestrator/ + # eagle-eye. A single aggregated endpoint needs no cross-rank KV store, no global KV + # index and no KV-cache-aware routing — matches the qwen3-32b-agg-base / GLM paradigm + # (random routing, no indexer). The store is out on purpose (see the header). + infernex.io/disabled-components: "mooncake,cache-indexer,pd-orchestrator,eagle-eye" +spec: + model: + name: minimax-m2.5-w8a8 # == --served-model-name == MaaS modelName == client -m + # Weights are the node-local copies exposed by the hostPath PV+PVC above, mounted at + # /mnt/models; the storageInitializer is OFF (weights are already on-node). `uri` is a + # required CRD placeholder here. + # ALT (cross-environment): import the OCI ModelCar tar into your own registry once — + # skopeo copy --all \ + # oci-archive:MiniMax-M2.5-w8a8-QuaRot.oci.tar:v0.1.0 \ + # docker:///modelcar-minimax-m2.5-w8a8-quarot:v0.1.0 + # then set `uri: oci:///modelcar-minimax-m2.5-w8a8-quarot:v0.1.0`. + # Because this ModelCar is >200 GB, keep `storageInitializer.enabled: false` and add + # an init container that extracts the pulled layout to tmpfs (a plain storageInitializer + # pull needs ~2× the weight size in ephemeral disk). + uri: pvc://minimax-n0/ + replicas: 1 # ONE aggregated service (leader + worker below) = 16 cards + parallelism: + data: 2 # DP2 across the two nodes + dataLocal: 1 # 1 DP rank per node + storageInitializer: + enabled: false # weights already staged on-node (pvc:// hostPath) + # ===== router: hermes-router (EPP), random-picker. DP2 has one leader endpoint, so + # random is effectively a no-op, but the structure is kept for consistency. ===== + router: + gateway: {} + route: {} + scheduler: + pool: + spec: + endpointPickerRef: + failureMode: FailOpen + group: "" + kind: Service + name: minimax-m25-w8a8-agg-epp-service + port: + number: 9002 + selector: + matchLabels: + app.kubernetes.io/name: minimax-m25-w8a8-agg + app.kubernetes.io/part-of: llminferenceservice + kserve.io/component: workload + leaderworkerset.sigs.k8s.io/worker-index: "0" # target the DP leader (rank 0) + targetPorts: + - number: 8000 + template: + containers: + - args: + - --pool-name + - minimax-m25-w8a8-agg-inference-pool + - --pool-namespace + - demo + - --pool-group + - inference.networking.k8s.io + - --zap-encoder + - json + - --v + - "1" + - --tracing + - "false" + - --metrics-endpoint-auth + - "false" + - --config-text + - |- + apiVersion: inference.networking.x-k8s.io/v1alpha1 + kind: EndpointPickerConfig + plugins: + - type: metrics-data-source + parameters: { scheme: "http", path: "/metrics", insecureSkipVerify: true } + - type: core-metrics-extractor + - type: random-picker + schedulingProfiles: + - name: default + plugins: + - pluginRef: random-picker + image: build-harbor.alauda.cn/3rdparty/openfuyao/hermes-router:26.6.0-rc.3 + imagePullPolicy: IfNotPresent + livenessProbe: + failureThreshold: 3 + grpc: { port: 9003, service: liveness } + periodSeconds: 10 + timeoutSeconds: 5 + name: main + ports: + - { containerPort: 9002, name: grpc, protocol: TCP } + - { containerPort: 9003, name: grpc-health, protocol: TCP } + - { containerPort: 9090, name: metrics, protocol: TCP } + readinessProbe: + failureThreshold: 3 + grpc: { port: 9003, service: readiness } + periodSeconds: 10 + timeoutSeconds: 5 + resources: + limits: { memory: 4Gi } + requests: { cpu: "1", memory: 1Gi } + securityContext: + allowPrivilegeEscalation: false + capabilities: { drop: ["ALL"] } + readOnlyRootFilesystem: true + runAsNonRoot: true + seccompProfile: { type: RuntimeDefault } + startupProbe: + failureThreshold: 30 + grpc: { port: 9003, service: liveness } + periodSeconds: 10 + timeoutSeconds: 5 + volumeMounts: + - { mountPath: /etc/ssl/certs, name: tls-certs, readOnly: true } + dnsPolicy: ClusterFirst + restartPolicy: Always + serviceAccountName: minimax-m25-w8a8-agg-epp-sa + terminationGracePeriodSeconds: 30 + volumes: + - name: tls-certs + secret: + secretName: '{{ ChildName .ObjectMeta.Name `-kserve-self-signed-certs` }}' + # ===== leader (DP rank 0, node 0, mounts the node-0 weights PVC) ===== + template: + containers: + - args: + - | + set +e # a fresh-start getent lookup can transiently fail; set -e would crash bash immediately + source /usr/local/Ascend/ascend-toolkit/set_env.sh 2>/dev/null || true + source /usr/local/Ascend/nnal/atb/set_env.sh 2>/dev/null || true + for _i in $(seq 1 60); do DP_ADDR=$(getent hosts "${LWS_LEADER_ADDRESS}" 2>/dev/null | awk '{print $1}' | head -1); [ -n "${DP_ADDR:-}" ] && break; echo "[minimax] waiting LWS_LEADER_ADDRESS dns ($_i)"; sleep 2; done + DP_ADDR=${DP_ADDR:-${LWS_LEADER_ADDRESS}} + echo "[minimax-leader] HCCL_IF_IP=${HCCL_IF_IP} DP_ADDR=${DP_ADDR} devices=$(ls /dev/davinci* 2>/dev/null | tr '\n' ' ')" + exec vllm serve /mnt/models \ + --host 0.0.0.0 --port 8000 \ + --served-model-name minimax-m2.5-w8a8 \ + --seed 1024 \ + --trust-remote-code \ + --api-server-count 1 \ + --data-parallel-size 2 \ + --data-parallel-size-local 1 \ + --data-parallel-address ${DP_ADDR} \ + --data-parallel-rpc-port 13389 \ + --tensor-parallel-size 8 \ + --enable-expert-parallel \ + --quantization ascend \ + --enable-auto-tool-choice \ + --tool-call-parser minimax_m2 \ + --reasoning-parser minimax_m2 \ + --max-model-len 196608 \ + --max-num-batched-tokens 16384 \ + --max-num-seqs 32 \ + --gpu-memory-utilization 0.85 \ + --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":128}' \ + --async-scheduling \ + --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ + --additional-config '{"enable_cpu_binding":true}' + command: ["/bin/bash", "-c"] + env: + - { name: HOME, value: /home } + - { name: VLLM_LOGGING_LEVEL, value: INFO } + - { name: HF_HUB_OFFLINE, value: "1" } + - { name: HCCL_OP_EXPANSION_MODE, value: AIV } + - { name: HCCL_BUFFSIZE, value: "512" } + - { name: VLLM_ASCEND_ENABLE_FLASHCOMM1, value: "1" } # vendor minimax default; TP comm optimization (Reduce-Scatter + All-Gather) + - name: HCCL_IF_IP + valueFrom: { fieldRef: { fieldPath: status.podIP } } + - { name: GLOO_SOCKET_IFNAME, value: eth0 } + - { name: TP_SOCKET_IFNAME, value: eth0 } + - { name: HCCL_SOCKET_IFNAME, value: eth0 } + - { name: VLLM_RPC_TIMEOUT, value: "360000" } + - { name: VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, value: "3000" } + - { name: HCCL_EXEC_TIMEOUT, value: "200" } + - { name: HCCL_CONNECT_TIMEOUT, value: "120" } + - { name: OMP_PROC_BIND, value: "false" } + - { name: OMP_NUM_THREADS, value: "1" } + - { name: PYTORCH_NPU_ALLOC_CONF, value: expandable_segments:True } + - { name: ACL_OP_INIT_MODE, value: "1" } + - { name: TASK_QUEUE_ENABLE, value: "1" } + - { name: CPU_AFFINITY_CONF, value: "1" } + - { name: VLLM_ENGINE_READY_TIMEOUT_S, value: "1200" } + - { name: HCCL_INTRA_ROCE_ENABLE, value: "1" } # cross-pod EP all-to-all over NIC RoCE (DP2 × TP8 EP16, cross-node) + image: quay.io/ascend/vllm-ascend:v0.23.0-openeuler + imagePullPolicy: IfNotPresent + livenessProbe: + failureThreshold: 8 + httpGet: { path: /health, port: http } + periodSeconds: 15 + timeoutSeconds: 10 + name: main + ports: + - { containerPort: 8000, name: http, protocol: TCP } + readinessProbe: + failureThreshold: 3 + httpGet: { path: /health, port: http } + periodSeconds: 10 + resources: + limits: { cpu: "48", huawei.com/Ascend910: "8", memory: 480Gi } + requests: { cpu: "16", huawei.com/Ascend910: "8", memory: 96Gi } + securityContext: + allowPrivilegeEscalation: false + runAsGroup: 0 + runAsNonRoot: false + runAsUser: 0 + seccompProfile: { type: RuntimeDefault } + startupProbe: + failureThreshold: 160 # ~230 GB from a node-local mount + cross-node handshake + first graph compile: budget ~80 min + httpGet: { path: /health, port: http } + periodSeconds: 30 + volumeMounts: + - { mountPath: /mnt/models, name: model, readOnly: true } + - { mountPath: /dev/shm, name: shm } + hostIPC: true + tolerations: + - operator: Exists + volumes: + - name: model + persistentVolumeClaim: { claimName: minimax-n0 } # node-0 weights (leader) + - name: shm + emptyDir: { medium: Memory, sizeLimit: 32Gi } + # ===== worker (DP rank 1, node 1, mounts the node-1 weights PVC) ===== + worker: + containers: + - args: + - | + set +e + source /usr/local/Ascend/ascend-toolkit/set_env.sh 2>/dev/null || true + source /usr/local/Ascend/nnal/atb/set_env.sh 2>/dev/null || true + for _i in $(seq 1 60); do DP_ADDR=$(getent hosts "${LWS_LEADER_ADDRESS}" 2>/dev/null | awk '{print $1}' | head -1); [ -n "${DP_ADDR:-}" ] && break; echo "[minimax] waiting LWS_LEADER_ADDRESS dns ($_i)"; sleep 2; done + DP_ADDR=${DP_ADDR:-${LWS_LEADER_ADDRESS}} + echo "[minimax-worker] HCCL_IF_IP=${HCCL_IF_IP} DP_ADDR=${DP_ADDR} devices=$(ls /dev/davinci* 2>/dev/null | tr '\n' ' ')" + exec vllm serve /mnt/models \ + --host 0.0.0.0 --port 8000 \ + --headless \ + --served-model-name minimax-m2.5-w8a8 \ + --seed 1024 \ + --trust-remote-code \ + --data-parallel-size 2 \ + --data-parallel-size-local 1 \ + --data-parallel-start-rank 1 \ + --data-parallel-address ${DP_ADDR} \ + --data-parallel-rpc-port 13389 \ + --tensor-parallel-size 8 \ + --enable-expert-parallel \ + --quantization ascend \ + --enable-auto-tool-choice \ + --tool-call-parser minimax_m2 \ + --reasoning-parser minimax_m2 \ + --max-model-len 196608 \ + --max-num-batched-tokens 16384 \ + --max-num-seqs 32 \ + --gpu-memory-utilization 0.85 \ + --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":128}' \ + --async-scheduling \ + --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ + --additional-config '{"enable_cpu_binding":true}' + command: ["/bin/bash", "-c"] + env: + - { name: HOME, value: /home } + - { name: VLLM_LOGGING_LEVEL, value: INFO } + - { name: HF_HUB_OFFLINE, value: "1" } + - { name: HCCL_OP_EXPANSION_MODE, value: AIV } + - { name: HCCL_BUFFSIZE, value: "512" } + - { name: VLLM_ASCEND_ENABLE_FLASHCOMM1, value: "1" } + - name: HCCL_IF_IP + valueFrom: { fieldRef: { fieldPath: status.podIP } } + - { name: GLOO_SOCKET_IFNAME, value: eth0 } + - { name: TP_SOCKET_IFNAME, value: eth0 } + - { name: HCCL_SOCKET_IFNAME, value: eth0 } + - { name: VLLM_RPC_TIMEOUT, value: "360000" } + - { name: VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, value: "3000" } + - { name: HCCL_EXEC_TIMEOUT, value: "200" } + - { name: HCCL_CONNECT_TIMEOUT, value: "120" } + - { name: OMP_PROC_BIND, value: "false" } + - { name: OMP_NUM_THREADS, value: "1" } + - { name: PYTORCH_NPU_ALLOC_CONF, value: expandable_segments:True } + - { name: ACL_OP_INIT_MODE, value: "1" } + - { name: TASK_QUEUE_ENABLE, value: "1" } + - { name: CPU_AFFINITY_CONF, value: "1" } + - { name: VLLM_ENGINE_READY_TIMEOUT_S, value: "1200" } + - { name: HCCL_INTRA_ROCE_ENABLE, value: "1" } + image: quay.io/ascend/vllm-ascend:v0.23.0-openeuler + imagePullPolicy: IfNotPresent + livenessProbe: + exec: { command: ["bash", "-c", "pgrep -f 'vllm serve' >/dev/null"] } + failureThreshold: 6 + periodSeconds: 30 + timeoutSeconds: 10 + name: main + ports: + - { containerPort: 13389, name: dp-rpc, protocol: TCP } + resources: + limits: { cpu: "48", huawei.com/Ascend910: "8", memory: 480Gi } + requests: { cpu: "16", huawei.com/Ascend910: "8", memory: 96Gi } + securityContext: + allowPrivilegeEscalation: false + runAsGroup: 0 + runAsNonRoot: false + runAsUser: 0 + seccompProfile: { type: RuntimeDefault } + startupProbe: + exec: { command: ["bash", "-c", "pgrep -f 'vllm serve' >/dev/null"] } + failureThreshold: 160 + periodSeconds: 30 + volumeMounts: + - { mountPath: /mnt/models, name: model, readOnly: true } + - { mountPath: /dev/shm, name: shm } + hostIPC: true + tolerations: + - operator: Exists + volumes: + - name: model + persistentVolumeClaim: { claimName: minimax-n1 } # node-1 weights (worker) + - name: shm + emptyDir: { medium: Memory, sizeLimit: 32Gi } diff --git a/docs/en/inference_guide/index.mdx b/docs/en/inference_guide/index.mdx index 9754832..ee99611 100644 --- a/docs/en/inference_guide/index.mdx +++ b/docs/en/inference_guide/index.mdx @@ -19,8 +19,8 @@ KServe `LLMInferenceService` reconciled by the InferNex-Bridge into a load-aware (hermes-router / EPP) in front of the vLLM-Ascend instances. Most were run through the **same InferNex aggregation surface and the same two benchmark scenarios** (the two Qwen models share an identical 2 × TP=4 topology and are directly comparable; -DeepSeek-V4-Flash uses larger topologies, up to a 16-card DP2 × TP=8 cross-node -aggregation). For the runtime model (KServe, ModelCar storage, +DeepSeek-V4-Flash and MiniMax-M2.5 use larger MoE topologies, up to a 16-card +DP2 × TP=8 cross-node aggregation). For the runtime model (KServe, ModelCar storage, scheduling) see [Model Deployment & Inference](../model_inference/index.mdx). ## Validated models @@ -31,6 +31,7 @@ scheduling) see [Model Deployment & Inference](../model_inference/index.mdx). | Qwen3.6-27B (W8A8) | `qwen3_5` hybrid (GDN + MTP) | 27.78B | W8A8 | Ascend 910B4 ×8 | vLLM-Ascend nightly | [Qwen3.6-27B (W8A8)](./qwen3-6-27b-w8a8.mdx) | | DeepSeek-V4-Flash (W4A8) | `deepseek_v4` MoE (MLA + DSA + MTP) | 256-expert MoE | W4A8 | Ascend 910B4 ×8 | vLLM-Ascend nightly | [DeepSeek-V4-Flash (W4A8)](./deepseek-v4-flash-w4a8.mdx) | | DeepSeek-V4-Flash (W8A8) | `deepseek_v4` MoE (MLA + DSA + MTP) | 256-expert MoE | W8A8 | Ascend 910B3 ×16 | vLLM-Ascend nightly | [DeepSeek-V4-Flash (W8A8)](./deepseek-v4-flash-w8a8.mdx) | +| MiniMax-M2.5 (W8A8) | `minimax_m2` MoE (full-attn, 256-expert) | 230B-A10B | W8A8 (QuaRot) | Ascend 910B3 ×16 | vLLM-Ascend v0.23.0 | [MiniMax-M2.5 (W8A8)](./minimax-m2.5-w8a8.mdx) | The two Qwen models and DeepSeek-V4-Flash (W4A8) were validated on Ascend 910B4 (32 GB/card), driven through KServe `LLMInferenceService` with load-aware routing (InferNex-Bridge + @@ -44,6 +45,13 @@ two nodes = 16 cards** as a single **DP2 × TP=8 cross-node aggregation** with a **mooncake cross-rank KV store**, driven through both the internal KServe ingress and the product MaaS gateway. +MiniMax-M2.5 (W8A8, ~230 GB, 230B-A10B) was validated on the **same Ascend 910B3 +16-card DP2 × TP=8 cross-node aggregation**, but as a **plain `agg` baseline** — no +mooncake store and no speculative decoding. Because this model is "fast" (~10B activated ++ full attention + W8A8), both add-ons were measured to *lose*, so the production config +keeps only the decode graph + local prefix cache. It was additionally validated as a +**real coding agent** (OpenCode / Pi) through the product MaaS gateway. + ## Runtime images | Engine | Device | Image (validated tag) | Used by | Notes | @@ -51,6 +59,7 @@ the product MaaS gateway. | vLLM-Ascend v0.18.0 | Huawei Ascend NPU | `quay.io/ascend/vllm-ascend:v0.18.0-openeuler` | Qwen3-32B | Community vLLM for Ascend, V1 engine. Serves standard Qwen3 dense models directly. Match the tag's CANN version to your host NPU driver. | | vLLM-Ascend nightly (release-pinned) | Huawei Ascend NPU | `quay.io/ascend/vllm-ascend:nightly-releases-v0.22.1rc-openeuler` | Qwen3.6-27B (W8A8), DeepSeek-V4-Flash (W4A8) | Carries the `qwen3_5` Gated DeltaNet hybrid + MTP **and** the `deepseek_v4` MoE stack — the stock `v0.18.0` cannot load either. Use this **release-pinned** tag, **not** the moving `nightly-main-openeuler` (it drifted to a broken build whose TP workers crash). | | vLLM-Ascend nightly-main (dated snapshot) | Huawei Ascend NPU | `quay.io/ascend/vllm-ascend:nightly-main-openeuler-0709` | DeepSeek-V4-Flash (W8A8) | A **later** `nightly-main` snapshot (vLLM 0.23.0) that carries the DP2 cross-node graph + DSA-CP prefix-cache fixes the W8A8 cross-node deployment needs. Pin a **dated** snapshot like this one, not the moving `nightly-main` tag. | +| vLLM-Ascend v0.23.0 (release-pinned) | Huawei Ascend NPU | `quay.io/ascend/vllm-ascend:v0.23.0-openeuler` | MiniMax-M2.5 (W8A8) | The **release-pinned** `nightly-releases-v0.23.0-openeuler` build (vLLM 0.23.0), carrying the `minimax_m2` MoE stack + tool-call/reasoning parsers **and** vLLM #11505 (streaming tool-call arguments held until the closing ``). Prefer this dated **release** tag over the moving `nightly-main`. | :::tip The Ascend CANN images are arm64. Always match the runtime image's CANN version to @@ -63,8 +72,8 @@ are listed; other engines (MindIE, SGLang, …) were not benchmarked at this siz These models were measured with `aiperf` against the same two scenarios, modelled on real serving patterns. Output is pinned to **128 tokens** and load is **closed-loop, concurrency 4** (4 in-flight requests, fixed); each scenario ran 240 requests. -DeepSeek-V4-Flash (W8A8) is the exception — it was swept at **concurrency 8 / 16 / 32** -(480 requests per tier). +DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) are the exceptions — they were swept at +**concurrency 8 / 16 / 32** (480 requests per tier). | Scenario | What it models | Dataset | Request shape | |---|---|---|---| @@ -80,10 +89,11 @@ scales with the instance count. **TPS is the total-token (input + output) calibe decode-only output rate is reported separately and is much smaller under these long-input workloads. -DeepSeek-V4-Flash (W8A8) instead runs a **16-card DP2 × TP=8 cross-node** deployment and -is swept at **concurrency 8 / 16 / 32** (480 req/tier), so its numbers are an operating -envelope reported on its own page. It was driven through both the internal KServe -ingress and the product **MaaS gateway (API-key) ingress**. +DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) instead run a **16-card DP2 × TP=8 +cross-node** deployment and are swept at **concurrency 8 / 16 / 32** (480 req/tier), so +their numbers are an operating envelope reported on each model's own page. Both were +driven through both the internal KServe ingress and the product **MaaS gateway (API-key) +ingress**. ::: ## Deploy a validated model @@ -150,11 +160,13 @@ The three signed images and their digests: | `docker.io/alaudadockerhub/modelcar-deepseek-v4-flash-w4a8-mtp:v0.1.0` | `sha256:ef57a46726bec9c6505504262beaa55ec6c45d595a1726ab90d594f42606c6be` | :::note -DeepSeek-V4-Flash (**W8A8**) is not in this Cosign table — its ModelCar is distributed as -an **OCI Image Layout tar** (not a signed registry image), so there is no Cosign signature -to verify. Its integrity is content-addressed instead: the tar's `index.json` carries the -image digest and every blob under `blobs/sha256/` is verified against its own digest on -import (`skopeo copy` checks this). See its [page](./deepseek-v4-flash-w8a8.mdx). +DeepSeek-V4-Flash (**W8A8**) and MiniMax-M2.5 (**W8A8**) are not in this Cosign table — +their ModelCars are distributed as **OCI Image Layout tars** (not signed registry images), +so there is no Cosign signature to verify. Integrity is content-addressed instead: each +tar's `index.json` carries the image digest and every blob under `blobs/sha256/` is +verified against its own digest on import (`skopeo copy` checks this). See the +[DeepSeek-V4-Flash (W8A8)](./deepseek-v4-flash-w8a8.mdx) and +[MiniMax-M2.5 (W8A8)](./minimax-m2.5-w8a8.mdx) pages. ::: `--insecure-ignore-tlog=true` is required because these were signed with diff --git a/docs/en/inference_guide/minimax-m2.5-w8a8.mdx b/docs/en/inference_guide/minimax-m2.5-w8a8.mdx new file mode 100644 index 0000000..0e07b63 --- /dev/null +++ b/docs/en/inference_guide/minimax-m2.5-w8a8.mdx @@ -0,0 +1,260 @@ +--- +weight: 50 +i18n: + title: + en: MiniMax-M2.5 (W8A8) + zh: MiniMax-M2.5 (W8A8) +--- + +# MiniMax-M2.5 (W8A8) + +`minimax_m2`-architecture model (`MiniMaxM2ForCausalLM`) — a 256-expert MoE (8 experts +per token, **no** shared expert) with **62 layers of full attention** (GQA 48/8 heads, +head_dim 128 — a plain full-attention model, **not** a hybrid-KV / sliding-window one) +and a native MTP head, served as **W8A8** (`W8A8_DYNAMIC`, QuaRot int8 attention + +expert, no KV quant; ~230 GB weights, 71 shards, 62 layers). It is **230B total with +~10B activated per token**. The ~230 GB weights fit a single 8 × 64 GB node, but it was +validated on **Ascend 910B3 (64 GB/card) across two nodes = 16 cards** as a single +aggregated **DP2 × TP=8 + EP16** service — apples-to-apples with the DeepSeek-V4-Flash +16-card topology, with DP2 roughly doubling prefill throughput on the prefill-heavy +benchmark. It runs the vLLM-Ascend **v0.23.0** release engine through Alauda AI's +InferNex surface, and both benchmark scenarios were driven through the **MaaS gateway +(API-key) ingress** as a concurrency sweep (8 / 16 / 32). + +Unlike DeepSeek-V4-Flash (W8A8), MiniMax-M2.5 is served as a **plain aggregation with +no acceleration add-ons** — no mooncake KV store and no speculative decoding. That is +the deliberate production config: because this model is "fast" (~10B activated + full +attention + W8A8), both add-ons were measured to *lose* (see [Deployment spec](#deployment-spec)). + +## Model identity + +| Field | Value | +|---|---| +| Publisher | MiniMax | +| Architecture | `minimax_m2` (`MiniMaxM2ForCausalLM`) — 256-expert MoE (8 experts/token, no shared expert) + 62 layers **full attention** (GQA 48/8, head_dim 128; not hybrid-KV) + native MTP head; **230B total / ~10B activated** | +| Quantization | W8A8 (`W8A8_DYNAMIC`, **QuaRot**; int8 attention + expert, no KV quant; ~230 GB weights, 71 shards, 62 layers); MIT license | +| Context / vocab | 196,608 (~192k, `max_position_embeddings`, rope_theta 5e6) / 200,064 | +| Model source (W8A8) | https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot | +| ModelCar (OCI Image Layout tar) | `http://package-minio.alauda.cn:9199/packages/aml-models-oci/v0.1.0/MiniMax-M2.5-w8a8-QuaRot.oci.tar` | + +:::note +**The built-in MTP head is not usable on these weights.** The config declares +`use_mtp: true` / `num_mtp_modules: 3`, but the QuaRot-quantized checkpoint ships **no +MTP tensors** (the 143,967-tensor state dict stops at layer 61), so +`--speculative-config method=mtp` cannot start. MiniMax's only working speculative path +is **eagle3** (a separate draft model), and eagle3 was measured **out of the production +config** for this model — see [Deployment spec](#deployment-spec). +::: + +:::note +The W8A8 ModelCar is distributed as an **OCI Image Layout tar** on the internal package +store (link above), not a public Docker Hub image. The benchmark itself was run with the +weights staged on a **node-local PV/PVC per node**; that is the path the manifest below +mounts. The ModelCar is the same weights repackaged, for cross-environment distribution +— import it into your own registry once and repoint `model.uri` (see the **Deploy** +section). Because this ModelCar is **>200 GB**, an `oci://` storageInitializer pull needs +extra ephemeral-disk handling. +::: + +## Validated hardware × stack + +| Platform | Engine | Version / config | Status | +|---|---|---|---| +| Ascend 910B3 64 GB × 16, 2 nodes (1 instance, DP2 × TP=8 + EP16) | vLLM-Ascend | `v0.23.0-openeuler` (vLLM 0.23.0, release-pinned) | ✅ closed-loop, 2-scenario **concurrency sweep (8/16/32)**, 480 req/tier, 0 error; **plain agg baseline** (no spec decode, no mooncake); + real coding-agent validation (OpenCode / Pi), all via MaaS gateway | + +:::note +The `v0.23.0-openeuler` release engine carries the full `minimax_m2` stack (model + +`minimax_m2` tool-call / reasoning parsers) **and** vLLM #11505, which holds streaming +tool-call arguments until the closing `` so no trailing-tag fragment is +spliced into a command argument. It was chosen over the earlier `nightly-main-0709` +snapshot for that parser fix (validated over 3,123 tool executions with 0 tail +pollution); the tradeoff is a small decode regression vs 0709 — see +[Benchmark results](#benchmark-results). +::: + +## Model configuration + +| Parameter | Value | +|---|---| +| Tensor parallelism (`tensor-parallel-size`) | 8 | +| Data parallelism (`data-parallel-size`) | 2 (1 rank per node, cross-node) | +| Replicas (instances) | 1 aggregated service (leader + worker = 16 cards) | +| Expert parallelism (`enable-expert-parallel`) | on (EP16) | +| `max-model-len` | 196608 (agent long-context; benchmark scenarios ≤ ~17.5k) | +| `max-num-batched-tokens` | 16384 | +| `max-num-seqs` | 32 | +| `gpu-memory-utilization` | 0.85 (official value — long ctx, deliberately not maxed) | +| Quantization | `ascend` (W8A8_DYNAMIC / QuaRot) | +| Speculative decoding | **none** (built-in MTP absent in the QuaRot weights; eagle3 measured out — see below) | +| Decode graph | `FULL_DECODE_ONLY` (ITL ~34 ms vs eager's ~300 ms) | +| Prefix caching | enabled, **local** (~62.6 % hit on the real agent trace; 91.8–95.5 % Prompt-Token hit in the agent window) | +| Tool calling / reasoning | `--tool-call-parser minimax_m2` + `--enable-auto-tool-choice` + `--reasoning-parser minimax_m2` | +| Notable env | `VLLM_ASCEND_ENABLE_FLASHCOMM1=1`. **Do not** set `enable_fused_mc2` — it is "A3-recommended, not applicable to A2", and 910B3 is A2-class. | + +## Deployment spec \{#deployment-spec} + +Served as **`agg`** — a single DP2 × TP=8 aggregation with **no** mooncake KV store and +**no** speculative decoding. The router stays `random` and the cache-indexer is **off**: +a single aggregated endpoint needs no global KV index or KV-cache-aware routing (matching +the GLM-5.2 / Qwen3-32B aggregation paradigm). What is a net win — and what this config +keeps — is the `FULL_DECODE_ONLY` decode graph plus the engine's **local prefix cache**. + +| Component | `agg` | +|---|:--:| +| hermes-router (EPP) | ✅ started (single leader endpoint, `random` no-op) | +| Routing strategy | `random` | +| mooncake KV store | — (measured out; see below) | +| Speculative decoding (eagle3 / MTP) | — (measured out; see below) | +| cache-indexer / KV-cache-aware routing | — (not needed for a single aggregated endpoint) | +| Decode graph + local prefix cache | ✅ (the net-win acceleration this config keeps) | + +:::note +**Why no speculative decoding and no KV store — both were measured to lose on this +model.** Because MiniMax-M2.5 is "fast" (only ~10B activated + full attention + W8A8), +each add-on hurts: + +- **eagle3 (speculative decoding) — out, on two independent lines of evidence.** ① Batch + throughput is a net loss: scenario ① at concurrency 32 decodes **377 tok/s with eagle3 + vs 631 tok/s baseline** (the more speculative tokens, the worse). ② A **real 27k-prefill + agent request returns HTTP 500** (the engine stays up but every such request fails), + while baseline serves the identical request with 0 error. Mechanism: speculation trades + compute for latency — a net win only when decode is memory-bandwidth bound (single + stream), a net loss once compute saturates at batch; W8A8 base × full-precision eagle + draft also mismatch, giving only ~44 % pos-0 acceptance (0.56 accepted per draft). The + built-in MTP head is unusable (no MTP tensors in the quantized weights). +- **mooncake KV store — not worth it.** Even with a fully-warmed store (~91 % merged + prefix hit) it is a net loss: scenario ② throughput at concurrency 16 / 32 is **−14 % / + −13 %** with TTFT worse across the board. MiniMax prefill is so fast that re-fetching KV + over the network is ≥ recomputing it locally — the opposite of DeepSeek, where slow + prefill makes the store a clear win. On the real agent trace the **local** prefix cache + already covers 62.6 %, and the store adds only ~3.6 % on top. +::: + +## Deploy + +Self-contained InferNex manifest (engine inlined in the `LLMInferenceService` leader + +worker templates + hermes-router preset, DP2 × TP=8 across two nodes, plain `agg` +baseline): + +| Spec | File | +|---|---| +| agg, DP2 × TP=8 | [`minimax-m2.5-w8a8-agg-llmisvc.yaml`](https://github.com/alauda/aml-docs/tree/master/docs/en/inference_guide/assets/minimax-m2.5-w8a8/minimax-m2.5-w8a8-agg-llmisvc.yaml) | + +```bash +base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/inference_guide/assets/minimax-m2.5-w8a8 + +# 1. Stage the ~230 GB weights on a node-local directory on BOTH nodes (byte-identical), +# then edit the manifest: set the namespace, pin the two hostPath PVs to your node +# hostnames ( / ) and weights path, and set the image tag (match your +# CANN / host driver). Then apply — the engine mounts the on-node weights at /mnt/models +# (the storageInitializer is off, so there is no 230 GB pull). +kubectl apply -f $base/minimax-m2.5-w8a8-agg-llmisvc.yaml + +# Cross-environment alternative — import the OCI ModelCar tar into your own registry +# once (skopeo does the multi-arch amd64+arm64 push in one go), then set the manifest +# model.uri to oci:///modelcar-minimax-m2.5-w8a8-quarot:v0.1.0: +# curl -sO http://package-minio.alauda.cn:9199/packages/aml-models-oci/v0.1.0/MiniMax-M2.5-w8a8-QuaRot.oci.tar +# skopeo copy --all \ +# oci-archive:MiniMax-M2.5-w8a8-QuaRot.oci.tar:v0.1.0 \ +# docker:///modelcar-minimax-m2.5-w8a8-quarot:v0.1.0 +# (This ModelCar is >200 GB — keep storageInitializer.enabled:false and add an init +# container that extracts the pulled layout to tmpfs.) + +# Internal KServe ingress (no auth): +curl -s http:////minimax-m25-w8a8-agg/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"minimax-m2.5-w8a8","messages":[{"role":"user","content":"hello"}]}' + +# Product MaaS gateway (OpenAI-compatible, API-key auth + token rate limiting): +curl -s http:///v1/chat/completions \ + -H "Authorization: Bearer $MAAS_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{"model":"minimax-m2.5-w8a8","messages":[{"role":"user","content":"hello"}]}' +``` + +## Benchmark results \{#benchmark-results} + +Closed-loop `aiperf 0.7.0`, **DP2 × TP=8 (16 × 910B3 64 GB)**, driven through the product +**MaaS gateway**, on the production **`v0.23.0-openeuler`** engine. Concurrency sweep +**8 / 16 / 32**, 480 requests per tier (all tiers 0 error / 0 mismatch). TTFT / E2E in s, +ITL in ms; decode = output-only tok/s (measured), TPS = total tokens/s (input + output, +computed; prefill-dominated). Output is pinned to 128 tokens. + +**Scenario ① — fixed-length system-prompt reuse (ISL ~8k / OSL 128)** + +| Concurrency | TTFT avg (s) | ITL avg (ms) | E2E avg (s) | Decode (tok/s) | TPS (in+out) | +|--:|--:|--:|--:|--:|--:| +| 8 | 1.17 | 33.5 | 5.4 | 189 | 12,004 | +| 16 | 1.35 | 36.2 | 5.9 | 344 | 21,967 | +| 32 | 1.44 | 45.6 | 7.2 | 564 | **36,012** | + +**Scenario ② — multi-turn dialogue, long context (ISL ~17.5k / OSL 128)** + +| Concurrency | TTFT avg (s) | ITL avg (ms) | E2E avg (s) | Decode (tok/s) | TPS (in+out) | +|--:|--:|--:|--:|--:|--:| +| 8 | 2.95 | 44.0 | 8.5 | 120 | 17,001 | +| 16 | 3.81 | 56.7 | 11.0 | 186 | 26,225 | +| 32 | 5.42 | 87.9 | 16.6 | 247 | **34,906** | + +:::note +**Engine choice — `v0.23.0` vs the earlier `nightly-main-0709`.** Re-tested at the same +hardware / config / caliber, `v0.23.0` shows a **small decode regression** vs 0709 +(decode throughput **−3…11 %**, ITL **+5…18 %**, growing with concurrency; e.g. scenario +① conc 32 decode 631 → 564 tok/s), while **TTFT (prefill) is flat or better**. The +regression is accepted in exchange for a **release-pinned** (non-drifting) tag, a unified +`minimax_m2` parser, and the #11505 streaming-tool-call fix. The 0709 image is kept on the +nodes as a fallback for delivery cases that are decode-throughput sensitive. +::: + +**Config decisions — eagle3 and mooncake both lose (measured on the baseline)** + +| Config (scenario · conc) | Baseline | eagle3 | mooncake store | +|---|--:|--:|--:| +| S① conc 32 — decode (tok/s) | **631** | 377 (1-tok) / 300 (3-tok) | — | +| S① conc 32 — ITL (ms) | **39.5** | 75.7 / 97.7 | — | +| S② conc 16 — TPS (in+out) | **27,792** | — | 23,902 (**−14 %**) | +| S② conc 32 — TPS (in+out) | **37,458** | — | 32,421 (**−13 %**) | +| Real 27k-prefill agent request | 0 error | **HTTP 500** | — | + +→ The plain `agg` **baseline wins on every axis**; both acceleration layers backfire on +this fast model. (The eagle3 / mooncake A/B runs were done on the earlier 0709 baseline, +before the `v0.23.0` re-test; the decision holds under both engines.) + +:::note +**How to read these.** Every tier completed 480/480 with zero errors and zero output +mismatches. As concurrency rises, **system throughput (decode / TPS) climbs** (sub-linear, +saturating) while tail latency (TTFT / ITL / E2E) also climbs — the queueing cost of more +in-flight requests. TPS is the **total-token (input + output)** caliber and is +prefill-dominated (ISL 8k / 17.5k, OSL 128); the decode-only output rate is the separate +"Decode" column. + +These numbers are **not directly comparable** to the other models in this guide: it is a +concurrency sweep (not the fixed concurrency-4 run the Qwen models use), on **910B3 64 GB** +cards, with a **16-card DP2 × TP=8** topology. Treat them as the operating envelope of +this specific large-MoE deployment, not a cross-model ranking. +::: + +## Real coding-agent validation + +Beyond the synthetic aiperf sweep, MiniMax-M2.5 was driven as a **real coding agent** +through the MaaS gateway. The **Envoy AI Gateway v0.6.0** passes the vLLM Chat Completions +stream through OpenAI→OpenAI unchanged, so the `minimax_m2` parser's streaming +`reasoning_content` and structured tool calls reach the client intact. Two agents +(**OpenCode 1.17.18** and **Pi 0.80.6**) ran the same fixed 14-task Terminal-Bench set at +concurrency **1 / 8 / 32** (14 / 14 / 42 runs each), scored by an objective verifier. + +| Agent | Passed (conc 1 / 8 / 32) | Peak correct-task throughput | +|---|---|--:| +| OpenCode 1.17.18 | 6 / 14 · 4 / 14 · 14 / 42 | 24.65 tasks/hr | +| **Pi 0.80.6** | 7 / 14 · 8 / 14 · 18 / 42 | **34.98 tasks/hr** | + +- **The old `` command-tail pollution is fixed:** across 3,123 real tool + executions, no execution argument contained a closing tag fragment (the reason the + engine is pinned to `v0.23.0` / #11505). +- **The link is fast and not saturated at conc 32:** 0 waiting requests, KV-cache peak + 14.6 %, 16-card AICore avg/p95 17.2 % / 34 %, vLLM 0 error / abort / preemption — so + 32 is a validated lower bound, not a capacity ceiling. +- **Residual parser risk remains on complex chains:** OpenCode mis-parsed raw tool XML as + text/reasoning in **10 / 70** trials; Pi merged adjacent tool boundaries in **2 / 70**. + The verdict is therefore **usable for POC, not yet claimed production-stable**, and Pi + is the stronger of the two agents on this model.