I recently got a compute module 4 (CM4) donated to add support for NVMe in raspiBackup (See here for details). I'm used to use a Raspberry but never used a CM4 before so I had to ramp up on CM4 first. The following page describes how I managed to get a RaspbianOS (Buster) up and running on the CM4 and to boot from NVMe.
My CM4 specs: 1GB main memory, 32GB eMMC and 128GB NVMe from Hynix, no WiFi and a Waveshare Mini Base Board A (CM4-IO-BASE-A)
Docs about compute model 4 I read first
How I primed the compute model 4 with RaspbianOS on eMMC
Then I found Full Compute Module 4 (Raspberry Pi) Setup / Imaging Guide from James Chambers from 8/2/2021. I executed all the steps described there. I used the RaspberryOS lite version because I just need to run the CM4 headless and thus had to create file ssh in /boot. WiFi config was not required. i also used the default /boot/config.txt with no
Finally I was able to boot my CM4 from eMMc.
How I primed the compute model 4 with RaspbianOS on NVMe
Now I had to prime the NVMe device. I booted my CM4 in mass mode, mounted the discovered NVMe on a Linux box, downloaded a RaspberryOS Image and copied the image on the NVMe with dd.
That's how you I booted my CM4 in mass mode:
In the usbboot directory of the downloaded github repo I executed on my Linux box with CM4 USB boot mode on and the CM4 connected via a USB data cable
sudo ../rpiboot -d .
Then I was able to mount both eMMC storage and NVMe storage on my Linux box. Finally I copied the RaspbianOS with dd on the NVMe.
Note: There is no guarntee the storage is mounted to the same devices all the time. I got /dev/sde used for eMMC and /dev/sdf used for NMVe but I also got the devices swapped !
When I then booted the CM4 still the eMMC was used. Root cause is the default BOOT_ORDER which is 0xf25641 as default. The boot sequence is defined from right to left so SD card is tried first (#1). Because I have an eMMC and thus the SD card is a dummy the system boots from eMMC. Next definition (#4) USB mass storage is used (e.g. USB attached disk) and next (#6) NVMe boot is used. When the boot sequence is changed to 0xf25416 with NVMe first it will boot from NVMe.
Note: If you use the bootcode from msd subdirectory you will not be able to access NVMe but eMMC only.
How I changed the boot sequence to boot from NVMe
Go into the usbboot directory and edit recovery/boot.conf. Change the BOOT_ORDER line and move #6 to the right. On this page the contents and semantic of BOOT_ORDER is described in detail. Next execute
which will create a new pieeprom.bin. Finally connect the CM4 in USB boot mode to your Linux box and execute
../rpiboot -d .
This will update the boot sequence in EEPROM. Now turn off USB boot mode and your CM4 will boot from NVMe. If you want to boot from eMMC either change the BOOT_ORDER again or just unplug the NVMe.
If for some reasons CM4 doesn't boot from NVMe double check the current BOOT_ORDER is correct with