diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md new file mode 100644 index 000000000..ac662be03 --- /dev/null +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,463 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# Attach High-Performance SR-IOV VFs to DPDK Pods and KubeVirt VMs with Multus on ACP + +## Issue + +Users running DPDK/CNF Pods or KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs as high-performance service NICs or PCI passthrough devices through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to workloads, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. + +This article follows the user workflow for attaching SR-IOV VFs with Multus on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use `vfio-pci` SR-IOV VFs from either a DPDK/CNF Pod or a KubeVirt VM. + +## Environment + +This article applies to the following combination: + +| Component | Version or description | +| --- | --- | +| Alauda Container Platform | 4.3 | +| Plugin | `sriov-network-plugin` | +| Plugin package version | `sriov-network-plugin v4.3.8` | +| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | +| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | +| Multi-NIC base | ACP provides Multus capability; Pods or VMs reference the SR-IOV VF through NAD | + +This article covers installing and using the SR-IOV L5 plugin for DPDK/CNF Pod and KubeVirt VM VF attachment. It does not cover OVS-DPDK, Userspace CNI, or internal DPDK application parameters. + +## Prerequisites + +### Nodes and hardware + +Prepare at least one worker node that meets these requirements: + +- The node has a physical NIC PF that supports SR-IOV. +- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. +- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. + +On the node, use the following checks to confirm that IOMMU is enabled: + +```bash +cat /proc/cmdline +dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi' +``` + +`/proc/cmdline` usually contains a parameter such as `intel_iommu=on` or `amd_iommu=on`, and `dmesg` should include IOMMU initialization logs. After a VF is created, also confirm that the VF has an IOMMU group: + +```bash +readlink -f /sys/bus/pci/devices//iommu_group +``` + +If the path does not exist, the node usually does not have usable IOMMU isolation, and the `vfio-pci` or PCI passthrough scenario should not continue. + +The node kernel must also have VFIO support enabled. At minimum, confirm that the `vfio-pci` module is available: + +```bash +lsmod | grep vfio +modprobe vfio-pci +``` + +To load the module automatically after node reboot, write it to the system module-load configuration: + +```bash +echo vfio-pci > /etc/modules-load.d/vfio-pci.conf +``` + +### Label SR-IOV nodes + +`sriov-network-config-daemon` is scheduled only to nodes labeled with `feature.node.kubernetes.io/sriov-capable=true` by default. Before installing the plugin, label the nodes that actually have SR-IOV PFs: + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true --overwrite +``` + +Apply this label only to nodes where VFs will be configured. Nodes without this label do not run `sriov-network-config-daemon` and do not get a corresponding `SriovNetworkNodeState`. + +## Resolution + +### Install the plugin + +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.8`. Download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. + +1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.8`. +3. Download the `sriov-network-plugin.*.v4.3.8.tgz` package that matches the target platform architecture. +4. Follow the [Cluster Plugin](https://docs.alauda.cn/container_platform/4.3/extend/cluster_plugin) instructions in the ACP User Guide to upload the package and install `sriov-network-plugin v4.3.8` in the target business cluster. + +When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. + +### Confirm the Multus base + +SR-IOV networks are attached to DPDK/CNF Pods or KubeVirt VMs by Multus as secondary networks. Before configuring SR-IOV networks, confirm in **Platform Management -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks). The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. + +After installation, confirm that the operator and config-daemon are running: + +```bash +kubectl get pods -n cpaas-system | grep sriov-network +``` + +At minimum, the following workloads should be `Running`: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +If `sriov-network-config-daemon` is not visible, first confirm that the target node is labeled with `feature.node.kubernetes.io/sriov-capable=true`. + +### Confirm SR-IOV PFs on nodes + +A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a Pod or VM. + +First list the node states synchronized by the operator, and select the target node from the `NAME` column: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +If a node with an SR-IOV NIC is missing from the list, first check whether the node has the `feature.node.kubernetes.io/sriov-capable=true` label. The config-daemon collects PF status only on nodes that match this label. + +If you need a quick host-side check to confirm that a physical NIC advertises SR-IOV capability before looking at the operator state, inspect the PCI capabilities of the PF. Replace `af:00.3` with the target PF PCI address: + +```bash +lspci -s af:00.3 -vvv | grep -i capabilities +``` + +If the output includes `Single Root I/O Virtualization (SR-IOV)`, the physical NIC advertises SR-IOV capability, for example: + +```text +Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) +``` + +This check only proves that the hardware exposes SR-IOV capability. It does not mean that the operator already supports the NIC or that VFs have already been created. Continue with the driver, allowlist, and `SriovNetworkNodePolicy` checks. + +On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\n"}{end}' +``` + +The output format is ` `, for example: + +```text +p1p1 0000:3d:00.0 19e5 1822 +``` + +Record the PF name to use, such as `p1p1`. Later, the `SriovNetworkNodePolicy` specifies the target nodes, PF name, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools. + +### Optional: Handle NICs not in the default supported list + +If `sriov-network-config-daemon` keeps logging messages like the following, the operator is running, but the NIC PCI ID is not in the default supported list. The operator skips that PF: + +```text +DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: 'hinic' ... product: 'Hi1822 Family (4*25GE)'"} +IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} +``` + +Add the NIC to the `supported-nic-ids` allowlist. Plugin `v4.3.8` supports configuring extra NICs from the installation or upgrade form, so users do not need to write chart values manually. When installing or upgrading `sriov-network-plugin` from **Platform Management -> Marketplace -> Cluster Plugins**, add a row in the **Extra Supported SR-IOV NICs** table and fill in the PCI IDs confirmed on site: + +| Field | Description | Example | +| --- | --- | --- | +| Name | Custom NIC name, used only as the allowlist key | `Huawei_Hi1822` | +| Vendor ID | PCI vendor ID of the PF | `19e5` | +| PF Device ID | PCI device ID of the PF | `1822` | +| VF Device ID | PCI device ID of the VF | `375e` | + +These example values only show the required field format. They do not mean that Huawei Hi1822 is built in by default. NIC models and VF device IDs can differ between customer environments, so use the on-site `lspci` output as the source of truth. + +If the VF device ID is unknown, temporarily create one VF during a maintenance window and inspect it. In the following example, `0000:3d:00.0` is the PF PCI address: + +```bash +echo 1 > /sys/bus/pci/devices/0000:3d:00.0/sriov_numvfs +readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 +lspci -Dnn -s +``` + +When reading the result, use the `:` pair inside brackets in the `lspci -Dnn` output. For example: + +```text +0000:3d:01.0 Ethernet controller [0200]: Huawei Technologies Co., Ltd. Hi1822 Family Virtual Function [19e5:375e] (rev 45) +``` + +In this case, `19e5` is the vendor ID, and `375e` is the VF Device ID that should be entered in the form. + +For on-site validation only, you can temporarily patch the ConfigMap in an installed cluster and restart the operator and config-daemon: + +```bash +kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ + '{"data":{"Huawei_Hi1822":"19e5 1822 375e"}}' + +kubectl rollout restart deployment/sriov-network-operator -n cpaas-system +kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system +``` + +This patch will not survive a plugin upgrade or reinstall. Use the plugin installation or upgrade form for long-term configuration. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: + +```bash +echo 0 > /sys/bus/pci/devices//sriov_numvfs +``` + +Otherwise, config-daemon may report that the PF already has VFs that were not created by the sriov operator and skip part of the change flow. + +### Configure and use an SR-IOV VF for DPDK/CNF Pods + +When a DPDK/CNF Pod needs a userspace process to own the VF directly, set `deviceType` to `vfio-pci`. In this mode, the VF is not used as a regular Linux network interface inside the Pod. Packet processing and address handling are owned by the application, and success is validated by a `/dev/vfio/` device in the container, not by a secondary interface such as `net1` in `ip a`. + +Save the following content as `sriov-node-policy-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-pod-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - + numVfs: 8 + deviceType: vfio-pci + mtu: 1500 +``` + +Replace `` with the PF name recorded earlier, such as `p1p1`. + +Apply the policy: + +```bash +kubectl apply -f sriov-node-policy-pod.yaml +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +If the output is a positive integer, such as `8`, the node has eight schedulable `openshift.io/sriov_vfio` VFs. + +If the resource does not appear in allocatable for a long time, check the target node synchronization status and error message: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `default`. Because the VF is owned directly by the DPDK/CNF application, Kube-OVN IPAM is not configured. + +Save the following content as `sriov-network-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: pod-sriov-net + namespace: cpaas-system +spec: + networkNamespace: default + resourceName: sriov_vfio +``` + +In this example, the operator generates a NAD named `pod-sriov-net` in the `default` namespace. If the application Pod runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Pod `metadata.namespace`, and the Pod NAD reference together. + +If the service network must use a specific VLAN, add `vlan: ` under `spec`. If this field is omitted, the network is untagged. The Pod default network is still provided by kube-ovn, and the SR-IOV VF is attached as a Multus secondary resource. + +Apply the objects and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network-pod.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n default pod-sriov-net +``` + +A Pod references the NAD through the `k8s.v1.cni.cncf.io/networks` annotation and requests one `vfio-pci` SR-IOV VF through resource requests. + +Save the following content as `sriov-pod.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-pod + namespace: default + annotations: + k8s.v1.cni.cncf.io/networks: default/pod-sriov-net +spec: + containers: + - name: app + image: + resources: + requests: + openshift.io/sriov_vfio: "1" + limits: + openshift.io/sriov_vfio: "1" +``` + +The `default/pod-sriov-net` value uses the `/` format. `openshift.io/sriov_vfio` corresponds to `SriovNetworkNodePolicy.spec.resourceName` and `SriovNetwork.spec.resourceName`. + +Apply the Pod and confirm that it is scheduled to a node with VF resources: + +```bash +kubectl apply -f sriov-pod.yaml +kubectl get pod -n default sriov-pod -o wide +``` + +Confirm that the container has received the VFIO device: + +```bash +kubectl exec -n default sriov-pod -- ls -l /dev/vfio/ +``` + +The expected output includes the `vfio` control device and one numeric IOMMU group device, for example: + +```text +crw------- 1 root root 234, 191 ... 191 +crw-rw-rw- 1 root root 10, 196 ... vfio +``` + +The numeric device, such as `191`, is the VFIO group available to the current Pod. + +To confirm the VF driver binding state, run `dpdk-devbind.py -s` on the host node. When the target VF appears under `Network devices using DPDK-compatible driver` with `drv=vfio-pci`, the VF is bound to a DPDK-compatible driver: + +```text +Network devices using DPDK-compatible driver +============================================ +0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +... +``` + +If the workload uses Deployment, StatefulSet, or another controller, set the NAD reference in the Pod template `metadata.annotations`, and request `openshift.io/sriov_vfio` in the application container `resources.requests` and `resources.limits`. DPDK applications usually also need CPU, HugePages, and startup parameters; the exact values are determined by the business image and application documentation. A reference Pod fragment is: + +```yaml +resources: + requests: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" + limits: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" +volumeMounts: + - name: hugepage + mountPath: /dev/hugepages +volumes: + - name: hugepage + emptyDir: + medium: HugePages +``` + +### Configure and use an SR-IOV network for KubeVirt VMs + +KubeVirt SR-IOV PCI passthrough passes the VF to the VM as a PCI device, so it also uses `deviceType: vfio-pci`. + +If a `SriovNetworkNodePolicy` with `resourceName: sriov_vfio` already exists for the same PF, skip this policy step and continue with the VM `SriovNetwork`. Create the following policy only when there is no reusable policy or the VM needs an independent VF pool. + +Save the following content as `sriov-node-policy-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vm-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - + numVfs: 8 + deviceType: vfio-pci + mtu: 1500 +``` + +Replace `` with the PF name recorded earlier, such as `p1p1`. + +If you created the policy above, apply it and confirm the node resource: + +```bash +kubectl apply -f sriov-node-policy-vm.yaml +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt`. If the VF is later owned by a DPDK application inside the VM, Kube-OVN IPAM is not required. + +Save the following content as `sriov-network-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vfio +``` + +In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, VM `metadata.namespace`, and the VM NAD reference together. + +If the service network must use a specific VLAN, add `vlan: ` under `spec`. If this field is omitted, the network is untagged. The VM default network is still provided by kube-ovn, and the SR-IOV VF is attached as a Multus secondary device and passed through to the VM. + +Apply the objects and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network-vm.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +The VM secondary NIC is configured in two places in the VM template: `spec.template.spec.domain.devices.interfaces` defines the NIC type seen by the VM, and `spec.template.spec.networks` defines which Multus NAD the NIC attaches to. The `name` values in both lists must match. + +Add both `interfaces[].sriov` and `networks[].multus` to the VM template to attach the SR-IOV secondary NIC. The following example shows only the network and interface-related fields: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. + +After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +Inside the guest operating system, confirm that the passed-through SR-IOV VF appears. Whether to bind the VF to DPDK, how to allocate HugePages inside the VM for the DPDK application, and how to configure EAL parameters or the service process are determined by the guest operating system and DPDK application documentation. diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md new file mode 100644 index 000000000..d87e6e566 --- /dev/null +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,463 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# 如何在 ACP 上通过 Multus 和 SR-IOV 为 DPDK Pod 和 KubeVirt 虚拟机挂载高性能 VF + +## 问题 + +用户在 Alauda Container Platform 4.3 上运行 DPDK/CNF Pod 或 KubeVirt 虚拟机,并希望通过 Multus 挂载宿主机 SR-IOV VF 作为高性能业务网卡或 PCI 直通设备。集群主 CNI 仍然可以是 kube-ovn;Multus 负责把辅助网络接入工作负载,SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源,并生成 Multus 使用的 `NetworkAttachmentDefinition`。 + +本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的 VF 接入:安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD,并分别说明 DPDK/CNF Pod 和 KubeVirt 虚拟机如何使用 `vfio-pci` 类型的 SR-IOV VF。 + +## 环境 + +本文适用于以下组合: + +| 组件 | 版本或说明 | +| --- | --- | +| Alauda Container Platform | 4.3 | +| 插件 | `sriov-network-plugin` | +| 插件包版本 | `sriov-network-plugin v4.3.8` | +| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| 部署命名空间 | `cpaas-system`(通过 ACP 市场安装时) | +| 主 CNI | 可使用 kube-ovn;SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | +| 多网卡基座 | ACP 已提供 Multus 能力,Pod 或虚拟机通过 NAD 引用 SR-IOV VF | + +本文覆盖 SR-IOV L5 插件在 DPDK/CNF Pod 和 KubeVirt 虚拟机 VF 接入场景中的安装与使用;不覆盖 OVS-DPDK、Userspace CNI 或 DPDK 应用内部参数配置。 + +## 先决条件 + +### 节点和硬件 + +至少需要一个满足以下条件的 worker 节点: + +- 节点上有支持 SR-IOV 的物理网卡 PF。 +- BIOS 和操作系统已经启用 IOMMU,例如 Intel VT-d 或 AMD-Vi。 +- PF 驱动支持创建 VF,且 PF 未被主 CNI 或 OVS 占用到无法配置 VF。 + +在节点上可以通过以下方式确认 IOMMU 已启用: + +```bash +cat /proc/cmdline +dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi' +``` + +`/proc/cmdline` 中通常应包含 `intel_iommu=on` 或 `amd_iommu=on` 等参数;`dmesg` 中应能看到 IOMMU 初始化相关日志。创建 VF 后,还可以确认 VF 存在 IOMMU group: + +```bash +readlink -f /sys/bus/pci/devices//iommu_group +``` + +如果该路径不存在,通常表示当前节点没有可用的 IOMMU 隔离,`vfio-pci`/PCI 直通场景不应继续配置。 + +节点内核还需要启用 VFIO 相关能力。至少确认 `vfio-pci` 模块可用: + +```bash +lsmod | grep vfio +modprobe vfio-pci +``` + +如果需要节点重启后自动加载,可将模块写入系统模块加载配置: + +```bash +echo vfio-pci > /etc/modules-load.d/vfio-pci.conf +``` + +### 标记 SR-IOV 节点 + +`sriov-network-config-daemon` 默认只调度到带有 `feature.node.kubernetes.io/sriov-capable=true` 标签的节点。安装插件前,先给实际带 SR-IOV PF 的节点打标签: + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true --overwrite +``` + +只给计划配置 VF 的节点打这个标签。没有该标签的节点不会运行 `sriov-network-config-daemon`,也不会生成对应的 `SriovNetworkNodeState`。 + +## 解决方案 + +### 安装插件 + +该能力作为 ACP 4.3 新功能交付,插件包版本为 `sriov-network-plugin v4.3.8`。先从 AC 应用市场下载插件包,再上传到目标 ACP 平台安装。 + +1. 登录 AC 应用市场,搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.8` 的安装包。 +3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.8.tgz` 包。 +4. 按照 ACP 用户手册中的[集群插件](https://docs.alauda.cn/container_platform/4.3/extend/cluster_plugin)说明上传插件包,并将 `sriov-network-plugin v4.3.8` 安装到目标业务集群。 + +通过 ACP 市场安装时,SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 + +### 确认 Multus 基座可用 + +SR-IOV 网络通过 Multus 作为辅助网络接入 DPDK/CNF Pod 或 KubeVirt 虚拟机。配置 SR-IOV 网络之前,先在 **平台管理 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装,先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成,不替代 Multus 元 CNI。 + +安装后确认 operator 和 config-daemon 已运行: + +```bash +kubectl get pods -n cpaas-system | grep sriov-network +``` + +预期至少看到以下工作负载处于 `Running`: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +如果看不到 `sriov-network-config-daemon`,先确认目标节点已经带有 `feature.node.kubernetes.io/sriov-capable=true` 标签。 + +### 确认节点上的 SR-IOV PF + +PF 是节点上的物理网卡,VF 是从 PF 创建出来并分配给 Pod 或虚拟机使用的虚拟 PCI 网卡。 + +先查看 operator 已同步的节点状态,并从输出的 `NAME` 列中选择目标节点: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +如果带 SR-IOV 网卡的节点没有出现在列表中,先检查该节点是否已经打上 `feature.node.kubernetes.io/sriov-capable=true` 标签;config-daemon 只会在该标签匹配的节点上采集 PF 状态。 + +如果需要先在宿主机侧快速判断某张物理网卡是否具备 SR-IOV 能力,可以直接查看该 PF 的 PCI capability。以下命令中,`af:00.3` 替换为目标 PF 的 PCI 地址: + +```bash +lspci -s af:00.3 -vvv | grep -i capabilities +``` + +如果输出中包含 `Single Root I/O Virtualization (SR-IOV)`,说明这张物理网卡声明了 SR-IOV capability,例如: + +```text +Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) +``` + +这个检查只说明硬件具备 SR-IOV capability,不等于 operator 已经支持或已经创建出 VF;后续仍需继续确认 driver、白名单和 `SriovNetworkNodePolicy` 配置。 + +在带 SR-IOV PF 的节点上,确认 operator 能发现物理网卡: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\n"}{end}' +``` + +输出格式为 ` `,例如: + +```text +p1p1 0000:3d:00.0 19e5 1822 +``` + +记录要使用的 PF 名称,例如 `p1p1`。后续创建 `SriovNetworkNodePolicy` 时,会在 policy 中指定目标节点、PF 名称、VF 数量、`resourceName` 和 `deviceType`。DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy;只有需要独立 VF 池时,才需要创建不同的 policy。 + +### 处理不在默认支持列表中的网卡(可选) + +如果 `sriov-network-config-daemon` 日志持续出现类似以下信息,说明 operator 已经运行,但该网卡的 PCI ID 不在默认支持列表中,operator 会跳过该 PF: + +```text +DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: 'hinic' ... product: 'Hi1822 Family (4*25GE)'"} +IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} +``` + +这类场景需要把网卡加入 `supported-nic-ids` 白名单。插件 `v4.3.8` 支持通过安装或升级表单配置额外网卡,不需要手工编写 chart values。在 **平台管理 -> 市场 -> 集群插件** 中安装或升级 `sriov-network-plugin` 时,在“额外支持的 SR-IOV 网卡”表格中新增一行,并按现场确认的 PCI ID 填写: + +| 字段 | 说明 | 示例 | +| --- | --- | --- | +| 名称 | 自定义网卡名称,只用于生成白名单键名 | `Huawei_Hi1822` | +| Vendor ID | PF 的 PCI vendor ID | `19e5` | +| PF Device ID | PF 的 PCI device ID | `1822` | +| VF Device ID | VF 的 PCI device ID | `375e` | + +以上示例只说明字段格式,不表示插件默认内置 Huawei Hi1822。不同客户环境的网卡型号和 VF device ID 可能不同,应以现场 `lspci` 输出为准。 + +如果还不知道 VF device ID,可以在维护窗口中临时创建 1 个 VF 后查看。以下示例中的 `0000:3d:00.0` 是 PF 的 PCI 地址: + +```bash +echo 1 > /sys/bus/pci/devices/0000:3d:00.0/sriov_numvfs +readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 +lspci -Dnn -s +``` + +判断时以 `lspci -Dnn` 输出中方括号内的 `:` 为准。例如: + +```text +0000:3d:01.0 Ethernet controller [0200]: Huawei Technologies Co., Ltd. Hi1822 Family Virtual Function [19e5:375e] (rev 45) +``` + +其中 `19e5` 是 vendor ID,`375e` 才是要填写到表单中的 VF Device ID。 + +仅在现场验证时,可以临时 patch 已安装集群中的 ConfigMap,并重启 operator 和 config-daemon: + +```bash +kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ + '{"data":{"Huawei_Hi1822":"19e5 1822 375e"}}' + +kubectl rollout restart deployment/sriov-network-operator -n cpaas-system +kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system +``` + +这种 patch 在插件升级或重装后可能丢失;长期配置应使用插件安装或升级表单。若为了查看 VF ID 手动写过 `sriov_numvfs`,在让 operator 通过 `SriovNetworkNodePolicy` 接管前,先清理手工创建的 VF: + +```bash +echo 0 > /sys/bus/pci/devices//sriov_numvfs +``` + +否则 config-daemon 可能提示该 PF 已有 VF 但不是由 sriov operator 创建,并跳过部分变更流程。 + +### 为 DPDK/CNF Pod 配置并使用 SR-IOV VF + +DPDK/CNF Pod 需要由用户态进程直接接管 VF 时,`deviceType` 使用 `vfio-pci`。这种模式下 VF 不作为 Pod 内的普通 Linux 网卡使用,业务报文处理和地址配置由容器内应用负责;成功标准是容器内出现 `/dev/vfio/` 设备,而不是 `ip a` 出现 `net1`。 + +将以下内容保存为 `sriov-node-policy-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-pod-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - + numVfs: 8 + deviceType: vfio-pci + mtu: 1500 +``` + +将 `` 替换为前面记录的 PF 名称,例如 `p1p1`。 + +应用 policy: + +```bash +kubectl apply -f sriov-node-policy-pod.yaml +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +如果输出为正整数,例如 `8`,说明当前节点有 8 个可调度的 `openshift.io/sriov_vfio` VF。 + +如果资源长时间未出现在 allocatable 中,再查看目标节点的同步状态和错误信息: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +创建 `SriovNetwork`,由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `default`;VF 由 DPDK/CNF 应用直接接管,因此这里不配置 kube-ovn IPAM。 + +将以下内容保存为 `sriov-network-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: pod-sriov-net + namespace: cpaas-system +spec: + networkNamespace: default + resourceName: sriov_vfio +``` + +上例中 operator 会在 `default` 命名空间生成名为 `pod-sriov-net` 的 NAD。如果业务 Pod 运行在其他命名空间,需要同时替换 `SriovNetwork.spec.networkNamespace`、Pod `metadata.namespace` 和 Pod 中的 NAD 引用。 + +如果业务网络需要接入指定 VLAN,可在 `spec` 中增加 `vlan: `;不配置时流量不携带 VLAN tag(untagged)。Pod 的默认网络仍由 kube-ovn 提供,SR-IOV VF 作为 Multus 辅助资源接入。 + +应用后确认 NAD 已生成: + +```bash +kubectl apply -f sriov-network-pod.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n default pod-sriov-net +``` + +Pod 通过 `k8s.v1.cni.cncf.io/networks` annotation 引用 NAD,并通过 resource request 申请一个 `vfio-pci` 类型的 SR-IOV VF。 + +将以下内容保存为 `sriov-pod.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-pod + namespace: default + annotations: + k8s.v1.cni.cncf.io/networks: default/pod-sriov-net +spec: + containers: + - name: app + image: + resources: + requests: + openshift.io/sriov_vfio: "1" + limits: + openshift.io/sriov_vfio: "1" +``` + +其中 `default/pod-sriov-net` 使用 `/` 格式;`openshift.io/sriov_vfio` 与 `SriovNetworkNodePolicy.spec.resourceName` 和 `SriovNetwork.spec.resourceName` 对应。 + +应用后确认 Pod 被调度到有 VF 资源的节点: + +```bash +kubectl apply -f sriov-pod.yaml +kubectl get pod -n default sriov-pod -o wide +``` + +确认容器内已经拿到 VFIO 设备: + +```bash +kubectl exec -n default sriov-pod -- ls -l /dev/vfio/ +``` + +预期至少看到 `vfio` 控制设备和一个数字命名的 IOMMU group 设备,例如: + +```text +crw------- 1 root root 234, 191 ... 191 +crw-rw-rw- 1 root root 10, 196 ... vfio +``` + +其中数字设备(如 `191`)表示当前 Pod 可用的 VFIO group。 + +如需辅助确认 VF 驱动绑定状态,可在宿主机节点上运行 `dpdk-devbind.py -s`。当目标 VF 出现在 `Network devices using DPDK-compatible driver` 区域,且显示 `drv=vfio-pci` 时,表示该 VF 已绑定到 DPDK 兼容驱动: + +```text +Network devices using DPDK-compatible driver +============================================ +0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +... +``` + +如果业务使用 Deployment、StatefulSet 等控制器,在 Pod template 的 `metadata.annotations` 中设置 NAD 引用,并在业务容器的 `resources.requests` 和 `resources.limits` 中申请 `openshift.io/sriov_vfio`。DPDK 应用通常还需要 CPU、HugePages 和启动参数等配置,具体取值由业务镜像和应用文档决定。参考配置片段如下: + +```yaml +resources: + requests: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" + limits: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" +volumeMounts: + - name: hugepage + mountPath: /dev/hugepages +volumes: + - name: hugepage + emptyDir: + medium: HugePages +``` + +### 为 KubeVirt 虚拟机配置并使用 SR-IOV 网络 + +KubeVirt SR-IOV PCI 直通场景需要把 VF 作为 PCI 设备传递给虚拟机,因此同样使用 `deviceType: vfio-pci`。 + +如果同一个 PF 已经有 `resourceName: sriov_vfio` 的 `SriovNetworkNodePolicy`,跳过本段,直接创建虚拟机使用的 `SriovNetwork`。只有没有可复用 policy,或虚拟机需要独立 VF 池时,才创建以下 policy。 + +将以下内容保存为 `sriov-node-policy-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vm-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - + numVfs: 8 + deviceType: vfio-pci + mtu: 1500 +``` + +将 `` 替换为前面记录的 PF 名称,例如 `p1p1`。 + +如果创建了上述 policy,应用并确认节点资源: + +```bash +kubectl apply -f sriov-node-policy-vm.yaml +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +创建 `SriovNetwork`,由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`;如果 VF 后续由虚拟机内的 DPDK 应用接管,这里不配置 kube-ovn IPAM。 + +将以下内容保存为 `sriov-network-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vfio +``` + +上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD。如果虚拟机运行在其他命名空间,需要同时替换 `SriovNetwork.spec.networkNamespace`、VM `metadata.namespace` 和 VM 中的 NAD 引用。 + +如果业务网络需要接入指定 VLAN,可在 `spec` 中增加 `vlan: `;不配置时流量不携带 VLAN tag(untagged)。虚拟机默认网络仍由 kube-ovn 提供,SR-IOV VF 作为 Multus 辅助设备直通给虚拟机。 + +应用后确认 NAD 已生成: + +```bash +kubectl apply -f sriov-network-vm.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +虚拟机辅助网卡配置在 VM 模板的两处:`spec.template.spec.domain.devices.interfaces` 定义虚拟机内看到的网卡类型,`spec.template.spec.networks` 定义这张网卡连接到哪个 Multus NAD。两处的 `name` 必须一致。 + +在 VM 模板中同时增加 `interfaces[].sriov` 和 `networks[].multus`,即可为虚拟机挂载 SR-IOV 辅助网卡。以下示例只展示网络和接口相关字段: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +上例中 VM 和 NAD 都在 `kubevirt` 命名空间,因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD,使用 `/` 格式。 + +创建 VM 后,确认 virt-launcher Pod 被调度到有 VF 资源的节点,并检查 VMI 状态: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +进入虚拟机操作系统后,确认出现直通的 SR-IOV VF。后续是否把该 VF 绑定给 DPDK、如何在虚拟机内部为 DPDK 应用分配 HugePages、如何配置 EAL 参数和业务进程,由虚拟机内的业务系统和 DPDK 应用文档决定。