Once we get the DHCP part set up, here comes the exciting part - to actually install the Linux distro and boot the Pi using it instead of the vanilla 32-bit Raspberry Pi OS.

Now our Pi is still stuck at the initial screen, but don’t worry, as long as it is trying to read the tftp path we have made progress.

Enable TFTP on the Server Side

Here I will be using the Synology NAS server as the file system server.

Synology offers tftp support in file sharing directly. On the other hand, it’s also fairly straightforward to set up a TFTP server using a Linux server. One thing important is to remember the tftp root directory. That will be used in the next step to copy over the boot files to.

Under that directory create the Pi corresponding directory like

1

<tftp root>/123456789

123456789 refers to last 9 chars of serial no. Pi’s bootloader by default will load from there.

Create the TFTP boot directory

We need to first grab the Linux distro from an image.

Here I picked Ubuntu 20.04 LTS which is “officially” supported. Technically this is probably not the best option as the OS itself includes a lot of things that we may not actually need and the size is pretty bulky. On the other end the Alpine one is very skinny and lacks some of the tooling we may need initially. For raspberry, unfortunately each distro may be slightly different (in terms of boot) so while you are free to choose whatever you want, YMMV.

Flash the image into the SD card and we’ll get 2 partitions - boot and root.

In general, it’s a good idea to first boot the system using the SD card just in case there’s some essential setup we need to complete ahead of time. After that, the remaining Pi boards can replicate the same set of files with minor tweaks.

Raspberry Pi 4 is shipped with a flashable EEPROM and supports netbooting. However, the entire setup process is not that straightforward and it’s thus worth writing down all the pitfalls through the path especially when it involves a non-“native” Linux distribution.

One thing worth noting is that even though this post mentions “PXE” and so does the official document, the boot process isn’t entirely PXE compliant and thus regular PXE boot support setup process may not work at all or at least not directly.

Here’s a diagram of the network topology I use for the setup:

The Ubiquiti will also be serving as the DHCP server. The Synology NAS will be serving data via tftp (pxe boot) and nfs (post boot). Due to kernel limitation, my NAS cannot support overlayfs. But for newer version of NAS it might be possible.

Raspberry Netboot Sequence

The bootloader in EEPROM by default does not enable netboot so we’d need to enable that first.

Once enabled, upon powering up, Pi will first send a DHCP request to ask for the TFTP server location and verify if netboot is supported.

After that it will fetch the firmware (start4.elf), configs and kernel files (vmlinuz) from that tftp location. The bootloader has a config to specify the exact path for the current device. By default, that’s the last 9 chars of the serial number. This is important as certain config (like cmdline.txt) would tell the OS how to mount the root fs and thus needs to be separate.

The kernel will later be loaded which takes over the boot process and eventually loads the rest of the system distribution.

The official doc can be found here.

Flashing the EEPROM to Enable Netboot

With the basic knowledge equipped, we can continue by first enabling the netboot in bootloader.

To do that, we need to grab the updated version of bootcode.bin file which supports netboot and update the config.

This also requires using the raspberry cli binary vcgencmd.

Pitfall #1: vcgencmd can only work in native Raspberry Pi OS

Yes the vcgencmd binary does not work in Ubuntu or any other “supported” Linux distros, even if you compile from source. They would just silently fail. Technically it should be doable if they are shipped with the right “stuff” but I decide not to waste more time figuring out what the “stuff” is.

Here we need a small microSD card to flash the Raspberry Pi OS onto it and boot the system. For headless setup, don’t forget to touch ssh to create the file to enable ssh by default.

We all write shitty code.

This seems like a stupidly bold statement from a random person, especially when this person very likely writes shitter stuff than most others. After all, there’s a well known joke across the industry - that is, Jeff Dean does not write buggy code, and when he does, it’s the problem of the compiler. So at least we have one counter example.

In common sense, masters don’t write shit, and Linus Torvalds very likely nods in the line.

Well, technically speaking, before we criticize this non sense bold statement, we need to define what shit is. And here, I think shitty code is anything that does not live up to the expectation. Buggy code is shitty - because it doesn’t provide consistently correct results when people expect it to. However, in the real world, we tend to operate at a much higher level - we care more about contracts, functionalities, and behaviors, and less about the exact implementations. For example, when we use an API service, we tend to care about what features it can provide, the QPS it can support, and how easily we can use it - in essence, our expectation. We probably won’t care about whether it’s run on bare metal or some public cloud, whether the memory is manually managed, by compiler, or by garbage collection.

Does that then mean all services/software that fail to match expectation start with shitty code? Well not necessarily. Windows 9x was great at the time it was first introduced and enabled a whole industry. Yet as things evolved, a lot of the design issues started to get exacerbated and it eventually got completely replaced by NT kernel. I’m not trying to defend 9x and saying it was very well designed masterpiece by any means - it had some notorious kernel safety issues after all, but just using it as an example to show that even if some software had highlights, over time other factors could expose the shittiness inside - it’s just a matter of time. What I believe is that no matter how well the systems were designed and implemented at the beginning, they deteriorate over time. We cannot prevent things from turning into shit - we can only slow down the speed.

What makes things more complicated is that, often times, there are way too many factors that have impact on how we value software:

TL;DR;

I was able to “break” Orbi’s 2.4G backhaul fallback
and hence force it onto 5G.

So instead of this

I got this:

The solution isn’t clean. Basically telnet to the satellite and then

1
2

root@RBS20:/# config set wlg_sta_ssid=<original ssid>_disabled
root@RBS20:/# config commit

Then reboot.

More details and investigation

↓