FAQ

1. Q: How to install Mellanox ConnectX-5 OFED driver and update NIC firmware?

   A: To use Mellanox NIC, firstly install the OFED driver MLNX_OFED, and then update NIC Firmware.

   The key steps are:

   • Download the OFED driver

   • Install OFED driver:

     $ sudo mount -o ro,loop MLNX_OFED_LINUX-5.4-3.1.0.0-ubuntu20.04-aarch64.iso /mnt
     $ sudo /mnt/mlnxofedinstall --upstream-libs --dpdk
     

   • Update firmware after OFED installation:

     $ wget https://www.mellanox.com/downloads/MFT/mft-4.20.0-34-arm64-deb.tgz
     $ tar xvf mft-4.20.0-34-arm64-deb.tgz
     $ cd mft-4.20.0-34-arm64-deb/
     $ sudo ./install.sh
     $ sudo mst start
     $ sudo mlxfwmanager --online -u -d <device PCIe address>
     

2. Q: How to enable SR-IOV VFs for Mellanox ConnectX-5?

   A: First, ensure the OFED driver and firmware tools are installed by following the steps above. Next, use this guide to enable SR-IOV on your platform.

   The key steps are:

   • Enable SR-IOV in BIOS settings

   • Enable SR-IOV on NIC firmware and set maximum number of VFs to a non-zero number:

     $ sudo mst status # lists NIC under "MST Devices"
     $ sudo mlxconfig -d /dev/mst/mt4121_pciconf0 set SRIOV_EN=True NUM_OF_VFS=4 # replace /dev/mst/... with your MST Device
     

   • Reboot the machine

3. Q: How to update firmware for Intel 700 Series NICs?

   A: To update the NIC firmware, first install the NIC on an x86 host. Then, install the NVM Update Utility on the x86 host and follow the included readme.

   The key steps are:

   • Insert the NIC into an x86 host

   • Download the NVM Update Utility

   • Unzip the package & extract the Linux version:

     $ unzip 700Series_NVMUpdatePackage_v8_70.zip
     $ tar xvf 700Series_NVMUpdatePackage_v8_70_Linux.tar.gz
     

   • Run the NVM Update Utility:

     $ cd 700Series/Linux_x64/
     $ sudo ./nvmupdate64e
     

   • Press A to update all NICs, or specify a list of cards to update

   • Reboot the machine

4. Q: How to setup SSH keys?

   A: SSH keys can be setup on Ubuntu 20.04 by following this guide. For AWS EC2 instances, follow the documentation instead.

   The key steps for Ubuntu 20.04 are:

   • Check whether SSH key already exists in ~/.ssh/id_rsa.pub. If not, create new authentication key pairs for SSH:

     $ ssh-keygen
     Generating public/private rsa key pair.
     Enter file in which to save the key (/home/<remote_user>/.ssh/id_rsa):
     Created directory '/home/<remote_user>/.ssh'.
     Enter passphrase (empty for no passphrase):
     Enter same passphrase again:
     Your identification has been saved in /home/<remote_user>/.ssh/id_rsa
     Your public key has been saved in /home/<remote_user>/.ssh/id_rsa.pub
     

   • Copy SSH public key to Arm machine. remote_user is the user name on Arm machine and dut.arm.com is the FQDN of Arm machine:

     $ ssh-copy-id [email protected]
     

   • Make sure remote_user can login Arm machine from management node without entering password:

     $ ssh [email protected]
     

5. Q: Can Ansible use SSH keys protected with a passphrase?

   A: Yes, by using the ssh-agent utility. Follow these steps to set it up.

6. Q: How to setup DUT with isolated CPUs and 1G hugepages?

   The key steps are:

   • Edit variable GRUB_CMDLINE_LINUX_DEFAULT in file /etc/default/grub to reserve hugepages using parameters default_hugepagesz=1G hugepagesz=1G hugepages=4 and reserve isolated CPU cores using parameters isolcpus=1-3 nohz_full=1-3 rcu_nocbs=1-3 cpuidle.off=1 cpufreq.off=1. If a DUT will be both a controller and worker node, replace CPUs 1-3 with 1-4. Ensure the number of isolated CPUs does not exceed the core count on the DUT.

     $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub | grep -v "#"
     GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=4 isolcpus=1-3 nohz_full=1-3 rcu_nocbs=1-3 cpuidle.off=1 cpufreq.off=1"
     

   • Enable above kernel parameters by executing sudo update-grub and rebooting the system:

     $ sudo update-grub
     $ sudo reboot now
     

     For EC2 instances, the command to update grub is:

     $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
     

   • Make sure kernel parameters take effect after reboot

     $ cat /proc/cmdline
     BOOT_IMAGE=/vmlinuz-5.19.10-051910-generic-64k root=/dev/mapper/ubuntu--vg-ubuntu--lv ro default_hugepagesz=1G hugepagesz=1G hugepages=4 isolcpus=1-3 nohz_full=1-3 rcu_nocbs=1-3 cpuidle.off=1 cpufreq.off=1
     $ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
     4
     $ grep Cpus_allowed_list /proc/self/status
     Cpus_allowed_list:     0,4-191
     

7. Q: Why isolate CPUs with these parameters?

   A: The combination of the kernel parameters mentioned above ensure that the CPUs run only the desired application. They never process kernel RCU callbacks (rcu_nocbs), don’t generate scheduling ticks (nohz_full) when there is only one process running on them, and are isolated from running any processes other than the ones pinned to them (isolcpus). Additionally, the cpufreq and cpuidle subsystems are disabled.

8. Q: How to build HTML rendered document?

   A: Install following dependencies and run make doc:

   sudo apt-get update
   sudo apt-get install python3-venv libenchant-dev -y
   cd <nw_cra_workspace>/cnf-reference-arch
   make doc