Vault

Raspberry pi net boot guide

{_obsidian_pattern_tag_public}

My notes for using rpi 3B+, dnsmasq, and iSCSI, all on linux (no NAS).

Other guides:
* https://warmestrobot.com/blog/2021/6/21/raspberry-pi-network-boot-guide
* https://www.server-world.info/en/note?os=Ubuntu_22.04&p=iscsi&f=2

Get dnsmasq to serve tftp (once for all pis)

In dnsmasq config, add this:

log=dhcp
enable-tftp
tftp-root=/opt/dnsmasq/tftp
pxe-service=0,"Raspberry Pi Boot"
 dhcp-mac=set:net-booting-rpi,b8:27:eb:*:*:*
 dhcp-reply-delay=tag:net-booting-rpi,2

(optional- to check PXE) A net-booting rpi will log lines like this:

15:01:13 dnsmasq-dhcp: vendor class: PXEClient:Arch:00000:UNDI:002001
15:01:13 dnsmasq-dhcp: DHCPDISCOVER(eth1) b8:27:eb:ab:cd:ef
15:01:13 dnsmasq-dhcp: reply delay: 2
15:01:16 dnsmasq-dhcp: tags: net-booting-rpi, eth1
15:01:16 dnsmasq-dhcp: requested options: 43:vendor-encap, 60:vendor-class, 67:bootfile-name,
15:01:16 dnsmasq-dhcp: requested options: 128, 129, 130, 131, 132, 133, 134, 135, 66:tftp-server,
15:01:16 dnsmasq-tftp: file /opt/dnsmasq/tftp/f63f14b6/start.elf not found for 10.2.0.110
15:01:16 dnsmasq-tftp: file /opt/dnsmasq/tftp/autoboot.txt not found for 10.2.0.110
...

Get rpi to do PXE boot (starting here the steps are per pi)

Needs rpi 3 B+ maybe. For k8s, use Pi{3A+,3B,3B+,...} for 64-bit.

On linux desktop:
1. Get rpi-imager (ubuntu package)
2. Insert a scratch SD card (at least 2GB)
3. sudo rpi-imager
4. OS: Raspberry Pi OS (other) -> Raspberry Pi OS Lite (64-bit)
5. Storage: (pick SD card)
6. Write.

Put SD card in pi and boot.
1. (some first-boot keys are generated, etc)
2. raspi-config
system -> hostname
that just runs
hostnamectl set-hostname "$NEW_HOSTNAME"
sed -i "s/127\.0\.1\.1.*$CURRENT_HOSTNAME/127.0.1.1\t$NEW_HOSTNAME/g" /etc/hosts
(and reboots)
Interfaces -> ssh -> yes
that just runs
ssh-keygen -Aupdate-rc.d ssh enableinvoke-rc.d ssh start`

  1. Check if the RPi 3B+ is configured already:
  2. (on pi) vcgencmd otp_dump | grep 17:
    17:1020000a
  3. The return value must be 17:3020000a

My Pi3B+ already were set. Pi3B needed program_usb_boot_mode=1 appended to /boot/firmware/config.txt then a reboot.

Set up iscsi target disk

On linux server (this is pyinfra code):

apt-get install -y tgt

iscsi_dir = '/d2/rpi-iscsi'
out= f'{iscsi_dir}/{pi_hostname}.disk'
files.directory(path=iscsi_dir)
server.shell(f'dd if=/dev/zero of={out} count=0 bs=1 seek=5G conv=excl || true')
files.put(dest=f"/etc/tgt/conf.d/{pi_hostname}.conf", src=StringIO(f"""
<target iqn.2024-03.com.bigasterisk:{pi_hostname}.target>
    backing-store {out}
    initiator-name iqn.2024-03.com.bigasterisk:{pi_hostname}.initiator
</target>
"""))
systemd.service(service='tgt.service', running=True, restarted=True)

(optional) Verify target is served

tgtadm --mode target --op show

Target 1: iqn.2024-03.com.bigasterisk:dlp.target01
    System information:
        Driver: iscsi
        State: ready
    LUN information:
        LUN: 0
            Type: controller
        LUN: 1
            Type: disk
            Size: 10 MB, Block size: 512

(optional) Test iscsi from any linux box:

  1. apt-get install -y open-iscsi
  2. Set /etc/iscsi/initiatorname.iscsi to:
    InitiatorName=iqn.2024-03.com.bigasterisk:ws-printer.initiator

  3. systemctl restart iscsid open-iscsi

  4. iscsiadm -m discovery -t sendtargets -p 10.2.0.133

    127.0.0.1:3260,1 iqn.2024-03.com.bigasterisk:dlp.target01
    ...
  1. iscsiadm -m node -o show
    node.name = iqn.2024-03.com.bigasterisk:dlp.target01
    node.conn[0].address = 127.0.0.1
    node.conn[0].port = 3260
    node.conn[0].startup = manual
  1. iscsiadm -m node --login -p 10.2.0.133
    Logging in to [iface: default, target: iqn.2024-03.com.bigasterisk:dlp.target01, portal: 127.0.0.1,3260]
    Login to [iface: default, target: iqn.2024-03.com.bigasterisk:dlp.target01, portal: 127.0.0.1,3260] successful.
  1. /
    tcp: [1] 127.0.0.1:3260,1 iqn.2024-03.com.bigasterisk:dlp.target01 (non-flash)
  1. cat /proc/partitions
    major minor  #blocks  name
    ...
       8       16      10240 sdb

Connect rpi to iSCSI

All commands on pi as root:

  1. apt-get install -y open-iscsi
    (this may do the update-initramfs described below- not sure)

  2. echo "InitiatorName=iqn.2024-03.com.bigasterisk:`hostname`" > /etc/iscsi/initiatorname.iscsi

  3. iscsiadm -m discovery -t sendtargets -p 10.2.0.133

```10.2.0.133:3260,1 iqn.2024-03.com.bigasterisk:ws-printer.target``` 3. ```iscsiadm -m node -l -T iqn.2024-03.com.bigasterisk:`hostname`.target -p 10.2.0.133``` Logging in to [iface: default, target: iqn.2024-03.com.bigasterisk:ws-printer.target, portal: 10.2.0.133,3260] Login to [iface: default, target: iqn.2024-03.com.bigasterisk:ws-printer.target, portal: 10.2.0.133,3260] successful. 1. `journalctl -n 100 | grep "Attached SCSI"` `Mar 27 22:33:28 ws-printer kernel: sd 0:0:0:1: [sda] Attached SCSI disk` 2. `mkfs.ext4 /dev/sda` ## Prep pi filesystem on SD before copying to tftp & iscsi All commands on pi as root: 1.
</div>
echo "ISCSI_INITIATOR=iqn.2024-03.com.bigasterisk:`hostname`.initiator" > /etc/iscsi/iscsi.initramfs
echo "ISCSI_TARGET_NAME=iqn.2024-03.com.bigasterisk:`hostname`.target" >> /etc/iscsi/iscsi.initramfs
echo 'ISCSI_TARGET_IP=10.2.0.133' >> /etc/iscsi/iscsi.initramfs
<div class="lang-general">
1.5. may need to udate /boot/firmware/cmdline.txt - not sure if that gets into the initramfs or what. 2. `update-initramfs -v -k $(uname -r) -c` ... `'/boot/initrd.img-6.6.20+rpt-rpi-v7' -> '/boot/firmware/initramfs7'` *or for 64bit* `'/boot/initrd.img-6.6.20+rpt-rpi-v8' -> '/boot/firmware/initramfs8'` ## Copy pi filesystem from SD to iscsi Optional: cull pi fs a ilttle: rm -r /var/swap /usr/share/{doc,man} All commands on pi as root: 1. `mount /dev/sda /mnt` 2. `rsync -ah --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/` 3. `mkdir /mnt/{dev,proc,sys,boot,mnt}` 4. Edit /mnt/etc/fstab to this:
</div>
proc            /proc           proc    defaults          0       0
/dev/sda        /               ext4    defaults,noatime  0       1
<div class="lang-general">
5. `umount /mnt` ## Serve the right tftp files The layout on pi /boot isn't quite what PXE boot will request. All commands on the dnsmasq host in `/opt/dnsmasq/tftp`. This could be shortened. "f63f14b6" is from pi's /proc/cpuinfo line `Serial : 00000000abcdef01`. 1. `rsync -aH drewp@ws-printer:/boot/ ./ws-printer-boot/` 2. (once for all pis) `ln -s ws-printer-boot/firmware/bootcode.bin bootcode.bin` 3. `mkdir f63f14b6` 4. `cp -a ws-printer-boot/firmware/* f63f14b6/` 5. edit f63f14b6/cmdline.txt to `console=serial0,115200 console=tty1 ip=::::ws-printer:eth0:dhcp root=/dev/sda ISCSI_INITIATOR=iqn.2024-03.com.bigasterisk:ws-printer.initiator ISCSI_TARGET_NAME=iqn.2024-03.com.bigasterisk:ws-printer.target ISCSI_TARGET_IP=10.2.0.133 ISCSI_TARGET_PORT=3260 rw rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory` (the cgroup keys are for k8s to work) ## Pre-Pi3B+ needs a bootcode SD card The rpi-imaged card already has FAT32 with bootcode.bin. Maybe that would work with rm'ing all but bootcode.bin. Didn't try. 1. `fdisk -l /dev/sde` Disk /dev/sde: 3.71 GiB, 3980394496 bytes, 7774208 sectors Disklabel type: dos Disk identifier: 0xed26035b Device Boot Start End Sectors Size Id Type /dev/sde1 * 2048 7774207 7772160 3.7G c W95 FAT32 (LBA) 2. `mkfs.fat -F 32 /dev/sde1` 3. `cp /tmp/bootcode.bin /mnt` 4. `umount /mnt` ## Reboot without SD card Done. ----- # no-SD-card process - trying alpine/qemu/nfs (and maybe overlayfs to share selected files between pis) - If it's Pi3B or earlier, we need bootcode.bin on SD card. Pi3B+ and newer needs nothing. 1. get `sudo kvm-ok` to pass. May need to go to bios and "Enter your BIOS setup and enable Virtualization Technology (VT)". Reboot is not enough (!); needs cold restart. ### notes on qemu ✨ dash(pts/1):~/pi-setup% qemu-img create -f qcow2 alpine.qcow2 2G Formatting 'alpine.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2147483648 lazy_refcounts=off refcount_bits=16 ✨ dash(pts/1):~/pi-setup% qemu-system-aarch64 -m 1G -nic user -boot d -kernel ~/Downloads/alpine-standard-3.19.1-aarch64.iso -hda alpine.qcow2 -display gtk -machine raspi3b qemu-system-aarch64: warning: netdev `{_obsidian_pattern_tag_net095}` has no peer qemu-system-aarch64: warning: requested NIC (anonymous, model unspecified) was not created (not supported by this machine?) ✨ dash(pts/2):~% fdisk -l Downloads/alpine-rpi-3.19.1-aarch64.img Disk Downloads/alpine-rpi-3.19.1-aarch64.img: 90 MiB, 94371840 bytes, 184320 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type Downloads/alpine-rpi-3.19.1-aarch64.img1 * 0 184463 184464 90.1M 6 FAT16