Changing Device Tree and Compiling Kernel for NVIDIA Jetson Xavier NX.

If you are designing a custom carrier board for the NVIDIA Jetson modules, chances are you will need to edit the device tree and recompile the kernel from source to bring up your board.

Given the poor documentation by NVIDIA, and my limited understanding of Linux inner workings, I have spent a lot of time gathering these info from the NVIDIA developer forums and getting my custom carrier board to work. 😫

For my case, I have designed my custom carrier board to be as close to the developer kit as possible. Still, there is a small difference in the HDMI Hot Plug Detect (HPD) polarity, and enabling SDMMC3 for using SD card.

My goal is to share the essence of these steps so that other engineers do not have to face the struggle that I have. The major steps are divided into the following sequence:

1. Set up cross compilation in Ubuntu
2. Setup the necessary files
3. Change pinmux (if applicable)
4. Modify device tree files (.dtsi)
5. Build kernel and device tree
6. Replace dtb files
7. Flash device tree

Set up cross compilation in Ubuntu

Since we are compiling for Jetson (ARM64 architecture) on our host machine which is most likely an x86 architecture, we need to download the corresponding compiler and set it up for cross-compilation. All the subsequent steps should work on Ubuntu 18 and 20.

NVIDIA recommends using the Linaro 7.3.1 2018.05 toolchain for this. Download the pre-built toolchain binaries here, and execute below:
Reference

mkdir $HOME/l4t-gcc
cd $HOME/l4t-gcc
tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

Setup the necessary files

Download Linux for Tegra BSP, BSP sources, sample root file system (RFS)
Note that you will need to create a NVIDIA developer account and login to download.

Extract kernel sources

In the public_sources.tbz2 (BSP sources zip file), there will be many other zip files inside, but we are only interested in kernel_src.tbz2. Extract this file into a folder to named “source”

Extract L4T and sample root file system

1. Extract “Linux_for_Tegra” from Jetson_Linux_R32.6.1_aarch64.tbz2 (it is 32.6.1 at the time of writing)
2. Extract sample RFS contents into Linux_for_Tegra/rootfs/

Create extlinux.conf

It is really weird that NVIDIA did not supply this file in the sample RFS!
You will need to create a directory called “extlinux” and create extlinux.conf in:

Linux_for_Tegra/rootfs/boot/extlinux/extlinux.conf

Copy the following contents into extlinux.conf:
Reference

TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk1p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0

Note that the reference link shows mmcblk0p1 but I have mmcblk1p1 here, this is because I am using a developer kit module with a custom carrier board with another sd card slot (SDMMC3). Since the developer kit module has an SD card slot (SDMMC1), we need to perform this fix, otherwise it would try to boot from the wrong SD card. I am not sure if we need to do this fix with the production module, you will need to try this out yourself.

Change Pinmux

If the pin functions defer from dev kit, changes to pinmux are most likely required. This step can only be done on windows since it makes use of Excel Macros

Download pinmux excel file.

Editing the excel file is pretty straightforward. For my case, I am using GPIO 08 for SD card detect pin. Since it was configured as cooling fan tachometer input on the development kit, I will need to change it into a regular GPIO.

Once done with your changes, click on the “Generate DT File” button, give your board a name, and you should see 3 files generated in the same folder.
Copy these files to your Ubuntu machine. In my case, I have named them “jetson_xavier_nx”

Generate .cfg files

These are the files that will be eventually flashed into Jetson, and here I took the shortcut of just replacing the original .cfg files meant for the developer kit, since my board is mostly similar to the developer kit. The proper method actually involves creating a new board name and a new set of files specific to that board.

The script to generate from .dtsi to .cfg files are located in:

Linux_for_Tegra/kernel/pinmux/python pinmux-dts2cfg.py

Copy the files into the same folder as the script, and run:
Reference

python pinmux-dts2cfg.py --pinmux addr_info.txt gpio_addr_info.txt por_val.txt --mandatory_pinmux_filemandatory_pinmux.txt tegra19x-jetson_xavier_nx_module-pinmux.dtsi tegra19x-jetson_xavier_nx_module-gpio-default.dtsi 1.0 > tegra19x-mb1-pinmux-p3668-a01.cfgpython pinmux-dts2cfg.py --pad pad_info.txt tegra19x-jetson_xavier_nx_module-padvoltage-default.dtsi  1.0 > tegra19x-mb1-padvoltage-p3668-a01.cfg

Copy the generated .cfg files and replace the files in:

Linux_for_Tegra/bootloader/t186ref/BCT/

Modify device tree files

I will show 2 examples here which is relevant for my case, this should give you an idea of how to tweak for other interfaces as well.

Changing the device tree requires editing the dtsi files, compiling them to dtb files, and flashing them to Jetson

In order to know which dtsi files to edit, log into a dev kit and do:

dmesg|grep dts

This will show us which folder we must go in the kernel sources. In Xavier NX’s case, its at:

source/hardware/nvidia/platform/t19x/jakku/kernel-dts/common

Change HDMI HPD polarity

Edit tegra194-p3509-disp.dtsi, change TEGRA_DC_OUT_HOTPLUG_LOW to TEGRA_DC_OUT_HOTPLUG_HIGH

Enable SDMMC3

Edit tegra194-p3688-common.dtsi to add sdmmc3 interface

sdmmc3: [email protected] {
mmc-ocr-mask = <0x0>;
cd-inverted;
cd-gpios = <&tegra_main_gpio TEGRA194_MAIN_GPIO(Q, 2) 0>;
nvidia,cd-wakeup-capable;
mmc-ocr-mask = <0>;
cd-inverted;
vmmc-supply = <&p3668_vdd_sdmmc3_sw>;
status = "okay";
};

Notice that TEGRA194_MAIN_GPIO(Q,2) came from pinmux.

There is also a load switch that enables the power supply to SD card, in my case, I have connected it to GPIO 03
Edit tegra194-fixed-regulator-p3668.dtsi and add the GPIO for controlling load switch:

p3668_vdd_sdmmc3_sw: [email protected] {
compatible = "regulator-fixed";
reg = <103>;
regulator-name = "vdd-sdmmc3-sw";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
gpio = <&tegra_main_gpio TEGRA194_AON_GPIO(CC, 0) 0>;
enable-active-high;
};

Notice TEGRA194_AON_GPIO(CC,0) came from pinmux:

Now why is it AON_GPIO instead of MAIN_GPIO? You can refer to this forum thread.
Also notice that I have written “[email protected]”, the “103” can really be any number as long as it doesn’t clash with other interfaces.
To find out which numbers are already being used, refer to this guide.

Build kernel and device tree

NVIDIA has provided a build script to build the kernel and device tree, at:

source/nvbuild.sh

But unfortunately, this script does not work at the point of writing 😑

Fix:
Change line 86 from:

O_OPT=(-O “${KERNEL_OUT_DIR}”)

to:

O_OPT=(O="${KERNEL_OUT_DIR}")

Set environment variable for compiler path:

export CROSS_COMPILE_AARCH64_PATH=“$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu”

Create a folder to store the compiled kernel & .dtb files (device tree blob). In my case I placed it at $HOME/Desktop/kernel_out

Execute:

./nvbuild.sh -o $HOME/Desktop/kernel_out/

If you interrupted the build process or did not do certain things correctly which causes the build to fail, you will be prompted with an error mentioning:
“please run make mrproper”

delete the following folder to fix this:
Reference

source/kernel/kernel-4.9/include/config/

Replace .dtb files

Copy tegra194-p3668-all-p3509–0000.dtb from

$HOME/Desktop/kernel_out/arch/arm64/boot/dts

Replace file in:

Linux_for_Tegra/kernel/dtb/

Flash device tree

At the time of writing, I am using jetson dev kit module on my custom carrier board, which uses an SD card on the module instead of a soldered 16gb emmc on the production module. Hence we first download the official xavier nx SD card image and flash it with balena etcher.

Insert the sd card into the module and perform the steps below to flash ONLY Device Tree:

We wont be changing the kernel here, and we will use whatever is in the default dev kit sd card image

1. Plug in micro usb to recovery port
2. Hold recovery button
3. Power on, and release recovery button
4. Flash device tree (in Linux_for_Tegra/)

sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit mmcblk0p1

5.In case connection fails, refer to this fix: