9. 使用Data Plane Development Kit (DPDK)测试网络环回
在开始Data Plane Development Kit (DPDK)之前，必须执行配置以太网接口中描述的配置步骤，因为FPGA要绑定到OPAE驱动程序(pac_n3000_net)才能开始DPDK。在使用DPDK期间，FPGA从此驱动程序解除了绑定，并绑定到vfio-pci驱动程序。
请按照以下步骤安装DPDK进行网络环回测试：
- 您必须使能主机上的Intel IOMMU驱动程序。请按照以下步骤使能Intel IOMMU驱动程序：
- 通过编辑/etc/default/grub将iommu=pt intel_iommu=on添加到GRUB_CMDLINE_LINUX entry。例如：
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/\ swap rhgb quiet pci=realloc intel_iommu=pt"
对于RHEL：还要将pci=realloc添加到GRUB_CMDLINE_LINUX entry。
- GRUB从传统的基于BIOS的机器上的/boot/grub2/grub.cfg文件或者从UEFI机器上/boot/efi/EFI/redhat/ grub.cfg文件读取其配置。根据您的系统，以root执行下面其中一个指令：
- 基于BIOS的机器：
grub2-mkconfig -o /boot/grub2/grub.cfg
- 基于UEFI的机器：
grub2-mkconfig -o /boot/efi/EFI/redhat/ grub.cfg
# grub2-mkconfig -o /boot/efi/EFI/redhat/ Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d Found initrd image: /boot/initramfs-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d.img /usr/sbin/grub2-mkconfig: line 290: /boot/efi/EFI/redhat/: Is a directory
- 基于BIOS的机器：
- 重启服务器以应用新的GRUB配置文件。
- 请运行以下命令来验证GRUB更新：
$ cat /proc/cmdline
下面的样例输出在内核命令行上显示intel_iommu=on。BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/rhel-root ro default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=2048 nosoftlockup mce=ignore_ce audit=0 isolcpus=1-11,24-35,13-23,36-47 nohz_full=1-11,24-35,13-23,36-47 rcu_nocbs=1-11,24-35,13-23,36-47 pci=realloc intel_iommu=on iommu=pt enforcing=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet skew_tick=1
- 通过编辑/etc/default/grub将iommu=pt intel_iommu=on添加到GRUB_CMDLINE_LINUX entry。例如：
- 安装所需的软件包：
$ sudo yum install readline-devel libpcap libpcap-devel numactl-devel
您需要下载这两个额外的Fedora软件包： 安装：$ sudo rpm -i <RPM file>
检查安装：$ rpm -qa | grep libfdt
- 编译和绑定驱动程序：
- 从DPDK社区下载DPDK代码，检查发布版本19.08。
$ git clone https://github.com/DPDK/dpdk.git
$ cd dpdk
$ git pull
Already up-to-date.
$ git reset --hard 31b798a6f08e9b333b94b8bb26910209aa810b73
HEAD is now at 31b798a build: avoid overlinking
$ wget https://patches.dpdk.org/series/6821/mbox/
Length: 179795 (176K) [text/plain] Saving to: \u2018index.html\u2019 100%[==============>] 179,795 554KB/s in 0.3s (554 KB/s) - \u2018index.html\u2019 saved [179795/179795]
$ git am index.html
Applying: net/i40e: i40e support ipn3ke FPGA port bonding Applying: raw/ifpga/base: add irq support Applying: raw/ifpga/base: clear pending bit Applying: raw/ifpga/base: add SEU error support Applying: raw/ifpga/base: add device tree support Applying: raw/ifpga/base: align the send buffer for SPI Applying: raw/ifpga/base: add sensor support Applying: raw/ifpga/base: introducing sensor APIs Applying: raw/ifpga/base: update SEU register definition Applying: raw/ifpga: add SEU error handler Applying: raw/ifpga: add PCIe BDF devices tree scan Applying: net/ipn3ke: remove configuration for i40e port bonding Applying: raw/ifpga/base: add secure support Applying: raw/ifpga/base: configure FEC mode Applying: raw/ifpga/base: clean fme errors Applying: raw/ifpga/base: add new API get board info Applying: raw/ifpga: add lightweight fpga image support Applying: raw/ifpga/base: add multiple cards support
$ vim config/common_linux // modify CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=n to CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
$ vim config/common_base // modify CONFIG_RTE_LIBRTE_IPN3KE_PMD=n to CONFIG_RTE_LIBRTE_IPN3KE_PMD=y
- 构建DPDK并导出RTE_SDK路径以指向dpdk目录：
注： 忽略消息“Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined"
$ export RTE_SDK=$PWD
$ export RTE_TARGET=x86_64-native-linuxapp-gcc
$ make config T=x86_64-native-linuxapp-gcc
$ make install -j8 T=x86_64-native-linuxapp-gcc
注： $RTE_SDK指向已解压的dpdk源位置。 - 如下所示，将FPGA和NIC绑定到vfio-pci驱动程序。下面的样例输出显示了2x2x25G配置的结果。8x10G配置产生类似的输出，输出差异如下表所示：
表 11. 输出差异 配置 每个Intel Ethernet Controller XL710-BM2 NIC的端口数 Intel Ethernet Controller XL710-BM2 NIC端口的器件ID 2x2x25G 2 0d58 8x10G 4 0cf8 $ cd $RTE_SDK
使用以下命令检查驱动程序与器件之间的绑定：$ ./usertools/dpdk-devbind.py --status-dev net
Sample output: Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:18:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em1 drv=tg3 unused= 0000:18:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em2 drv=tg3 unused= *Active* 0000:19:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em3 drv=tg3 unused= 0000:19:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em4 drv=tg3 unused= 0000:14:00.0 'Device 0d58' if=p1p1 drv=i40e unused= 0000:14:00.1 'Device 0d58' if=p1p2 drv=i40e unused= 0000:16:00.0 'Device 0d58' if=p1p3 drv=i40e unused= 0000:16:00.1 'Device 0d58' if=p1p4 drv=i40e unused= Other Network devices ===================== 0000:15:00.0 'Device 0b30' unused=intel_fpga_pci
安装vfio-pci kernel driver：$ sudo modprobe vfio-pci
将FPGA和FVL PF绑定到DPDK driver。将Bus:Device.Function替换成以上步骤中的输出。在8x10G映像的情况下，将所有的8 FVL B:D.F和FPGA绑定到vfio_pci：$ sudo ./usertools/dpdk-devbind.py -b vfio-pci 14:00.0 \ 14:00.1 15:00.0 16:00.0 16:00.1
返回./usertools/dpdk-devbind.py –-status-dev net检查FPGA和FVL PF是否绑定到vfio-pci driver。图 17. 样例输出
- 保留大页面：
$ sudo mkdir -p /mnt/huge
$ sudo mount -t hugetlbfs nodev /mnt/huge
$ sudo sh -c "echo 2048 > /sys/kernel/mm/hugepages/\ hugepages-2048kB/nr_hugepages"
- 从DPDK社区下载DPDK代码，检查发布版本19.08。
关于如何在DPDK中使能FPGA支持的详细信息，请参考Data Plane Development Kit Reference Manual: Intel FPGA Programmable Acceleration Card N3000 。