Rebuild the rootfs after SD loss - package everything that lived only on the card #21

New issue

Open

opened 2026-09-28 00:53:42 +00:00 by kit · 1 comment

kit commented

2026-09-28 00:53:42 +00:00

Owner

What happened

The SD card was wiped and reused by accident (2026-09-27), and there was no rootfs backup. A minimal replacement card now boots: the Aug 18 alarm/out/gts6l-alarm-rootfs.ext4 plus current modules, with the console on the serial jig. See MAINLINE-NOTES "Linux on the serial console". Everything set up by hand on the device after Aug 18 is gone. Every feature marked working in #5, #13, #16 and #18 depended on some of it.

What was lost

Inventory from baselines/good-boot-20260823-083110.state (a working boot, Aug 23) plus the notes:

Services (all enabled and active in the good boot):

rmtfs, pd-mapper, tqftpserv as proper units. Don't start them as background jobs from a oneshot: systemd reaps those (notes 2026-08-22).
gts6l-modem.service: starts the modem remoteproc. Writing start BLOCKS about 40 s, so run it in the background or raise TimeoutStartSec. It needs qcom_q6v5_pas.auto_boot=0 and recovery set to disabled BEFORE start (#16, commit b0c82aaf1d).
gts6l-wifi.service: only starts the modem, with Requires=/After=. pd-mapper has an empty WantedBy= and only came up through this chain.
gts6l-kmsgcap.service: dmesg to /root/kmsg.log. It's the log channel that survives resets (ramoops doesn't).
hexagonrpcd-sdsp.service: hexagonrpcd built from github linux-msm/hexagonrpc into /usr/local. It serves /usr/share/qcom/sm8150/Samsung/gts6l.
NetworkManager + wpa_supplicant, with the WiFi config.
systemd-timesyncd: /etc/systemd/timesyncd.conf.d NTP=pool.ntp.org. The RTC is read-only and reads 1970, which also breaks pacman's PGP checks.

Code / packages:

Patched rmtfs, source at /root/rmtfs. fd_open() falls back to the read-only shadow buffer on EROFS/EACCES, because fsg sits on a write-protected UFS LUN. Without it the modem gets no NV. It's upstreamable to linux-msm/rmtfs, so this time put it in a package or a fork.
libssc + iio-sensor-proxy (from ALARM extra).
The verbose hexagonrpcd debug build.

Config:

10-usb0.network.d/route.conf: a persistent default route via 172.16.42.2, metric 1000.
BT + MCFG firmware copied (not symlinked) into /usr/lib/firmware, working around the firmware-mount race at t=16 s.
A modules-load list for the q6 audio DAIs (#13).
The golden cmdline (from the good boot):
root=/dev/mmcblk0p2 rootwait rw console=tty0 consoleblank=0 earlycon=gtsfb,0x9c400000 panic=10 sysctl.kernel.panic_on_rcu_stall=1 ufs_qcom.dbg_rate_a=1 regulator_ignore_unused systemd.unit=multi-user.target
ufs_qcom.dbg_rate_a=1 is still required. Leaving it out offlines UFS and stalls init for 49 s. The serial variant swaps in earlycon=qcom_geni,0xa90000 console=ttyMSM0,115200n8.

What to do

Rebuild linux-gts6l (the kernel is 26 commits past its last package) and gts6l-device (the Aug 24 source was never built).
Move every item above into packages. Hand-written units and config go in gts6l-device. rmtfs, hexagonrpcd and libssc get their own PKGBUILDs.
Keep the boot-order caution from the notes: enabling rmtfs/pd-mapper/tqftpserv/gts6l-modem at boot once wedged the device. That was before the auto_boot=0 + recovery=disabled fix. Re-verify one service at a time, with the serial console attached.
Regenerate the rootfs with alarm/build-rootfs.sh extract && customize && image, and flash it with the golden cmdline.
Add a backup habit: a rootfs tarball or rsync of /etc + pacman -Qqe, pulled to the host.

Done when

A fresh card built only from packages boots with UFS, sensors (ssccli), WiFi, BT and the modem services all working together (Kit's in-tandem standard). Losing the card again would then cost one script run.

## What happened The SD card was wiped and reused by accident (2026-09-27), and there was no rootfs backup. A minimal replacement card now boots: the Aug 18 `alarm/out/gts6l-alarm-rootfs.ext4` plus current modules, with the console on the serial jig. See MAINLINE-NOTES "Linux on the serial console". Everything set up **by hand on the device** after Aug 18 is gone. Every feature marked working in #5, #13, #16 and #18 depended on some of it. ## What was lost Inventory from `baselines/good-boot-20260823-083110.state` (a working boot, Aug 23) plus the notes: **Services (all enabled and active in the good boot):** - `rmtfs`, `pd-mapper`, `tqftpserv` as **proper units**. Don't start them as background jobs from a oneshot: systemd reaps those (notes 2026-08-22). - `gts6l-modem.service`: starts the modem remoteproc. Writing `start` BLOCKS about 40 s, so run it in the background or raise TimeoutStartSec. It needs `qcom_q6v5_pas.auto_boot=0` and recovery set to `disabled` BEFORE start (#16, commit b0c82aaf1dbe). - `gts6l-wifi.service`: only starts the modem, with Requires=/After=. `pd-mapper` has an empty WantedBy= and only came up through this chain. - `gts6l-kmsgcap.service`: dmesg to `/root/kmsg.log`. It's the log channel that survives resets (ramoops doesn't). - `hexagonrpcd-sdsp.service`: hexagonrpcd built from github linux-msm/hexagonrpc into /usr/local. It serves `/usr/share/qcom/sm8150/Samsung/gts6l`. - NetworkManager + wpa_supplicant, with the WiFi config. - systemd-timesyncd: `/etc/systemd/timesyncd.conf.d` NTP=pool.ntp.org. The RTC is read-only and reads 1970, which also breaks pacman's PGP checks. **Code / packages:** - **Patched rmtfs, source at `/root/rmtfs`.** `fd_open()` falls back to the read-only shadow buffer on EROFS/EACCES, because fsg sits on a write-protected UFS LUN. Without it the modem gets no NV. It's upstreamable to linux-msm/rmtfs, so this time put it in a package or a fork. - libssc + iio-sensor-proxy (from ALARM extra). - The verbose hexagonrpcd debug build. **Config:** - `10-usb0.network.d/route.conf`: a persistent default route via 172.16.42.2, metric 1000. - BT + MCFG firmware **copied** (not symlinked) into /usr/lib/firmware, working around the firmware-mount race at t=16 s. - A modules-load list for the q6 audio DAIs (#13). - The golden cmdline (from the good boot): `root=/dev/mmcblk0p2 rootwait rw console=tty0 consoleblank=0 earlycon=gtsfb,0x9c400000 panic=10 sysctl.kernel.panic_on_rcu_stall=1 ufs_qcom.dbg_rate_a=1 regulator_ignore_unused systemd.unit=multi-user.target` **`ufs_qcom.dbg_rate_a=1` is still required.** Leaving it out offlines UFS and stalls init for 49 s. The serial variant swaps in `earlycon=qcom_geni,0xa90000 console=ttyMSM0,115200n8`. ## What to do 1. Rebuild `linux-gts6l` (the kernel is 26 commits past its last package) and `gts6l-device` (the Aug 24 source was never built). 2. Move **every** item above into packages. Hand-written units and config go in `gts6l-device`. rmtfs, hexagonrpcd and libssc get their own PKGBUILDs. 3. Keep the boot-order caution from the notes: enabling rmtfs/pd-mapper/tqftpserv/gts6l-modem at boot once wedged the device. That was before the auto_boot=0 + recovery=disabled fix. Re-verify one service at a time, with the serial console attached. 4. Regenerate the rootfs with `alarm/build-rootfs.sh extract && customize && image`, and flash it with the golden cmdline. 5. Add a backup habit: a rootfs tarball or rsync of /etc + `pacman -Qqe`, pulled to the host. ## Done when A fresh card built only from packages boots with UFS, sensors (`ssccli`), WiFi, BT and the modem services all working together (Kit's in-tandem standard). Losing the card again would then cost one script run.

kit added the

task

label

2026-09-28 00:53:42 +00:00

kit commented

2026-09-28 02:59:22 +00:00

Author

Owner

2026-09-27: offline half done - a rootfs built only from packages

Everything below is built and checked on the host. None of it has run on the device yet.

Packages (kit/gts6l-alarm, all in alarm/out/):

linux-gts6l 6.17.0-3: 43-patch series, incl. the MUIC UART fix and the #9 fuel gauge. The first time this PKGBUILD has ever built (it had two latent bugs: sources in a subdirectory, wrong checksum count).
gts6l-device 0.7: the whole Aug 23 on-device setup, rebuilt from the notes and #5. gts6l-rproc (auto_boot=0, recovery=disabled before start, modem held until uptime 120 s, configurable in /etc/gts6l/rproc.conf), a rmtfs drop-in -P -r -v (never -s: that stops the modem on exit, which is the TrustZone hang), sensors (persist ro,noload, served tree under /run, hexagonrpcd started by udev), BT/board.bin copied, kmsgcap, usb0 on networkd, NM for WiFi, timesyncd.
pd-mapper, tqftpserv, hexagonrpc: built natively in an aarch64 chroot. qrtr-git and rmtfs-git come from ALARM's aur repo; libssc and iio-sensor-proxy from extra.
gts6l-uboot 2025.10-2: U-Boot with menu 38 (SD boot + serial console) and menu 39 (XBL+ABL log to SD).

Rootfs: alarm/out/gts6l-alarm-rootfs.ext4 (4 GB, 2.2 GB used), via sudo WORK=<tmpfs> alarm/build-rootfs.sh extract && customize && image 4096. Verified by mounting it: all packages present, stock kernel removed, every Aug 23 unit enabled, /boot has our kernel + DTB, and resolv.conf / nsswitch.conf / pacman.conf are stock.

Deploy: tools/deploy-sd.sh /dev/sdX --write partitions the card, writes that image, and builds boot-gts6l.img + CRC from the rootfs's own kernel with the golden cmdline (tools/mkboot-gts6l.sh refuses any cmdline missing ufs_qcom.dbg_rate_a=1).

Deliberately NOT restored:

The patched rmtfs. Stock -r is enough (the modem never asked for EFS) and can't corrupt Samsung's NV. The patch is parked in alarm/upstream/rmtfs.

Still open, needs the tablet:

Deploy the card and boot it with the serial jig attached.
Verify each service in turn. The Aug 22 note says enabling the modem stack at boot once wedged the device; that was before the recovery=disabled fix.
The audio modules-load list is a best guess (the original was lost). Trim to what's actually needed.
Flash the new U-Boot with gts6l-flash-uboot and test menus 38/39.
Backup habit (step 5).

Bugs found along the way:

build-rootfs.sh copied resolv.conf through ALARM's absolute symlink onto the host's /run/systemd/resolve/resolv.conf. Fixed.
The old logsave/mirsave raw SD writes would land inside the new card's FAT. Fixed.

## 2026-09-27: offline half done - a rootfs built only from packages Everything below is built and checked on the host. **None of it has run on the device yet.** **Packages** (kit/gts6l-alarm, all in `alarm/out/`): - `linux-gts6l` 6.17.0-3: 43-patch series, incl. the MUIC UART fix and the #9 fuel gauge. The first time this PKGBUILD has ever built (it had two latent bugs: sources in a subdirectory, wrong checksum count). - `gts6l-device` 0.7: the whole Aug 23 on-device setup, rebuilt from the notes and #5. gts6l-rproc (auto_boot=0, recovery=disabled before start, modem held until uptime 120 s, configurable in `/etc/gts6l/rproc.conf`), a rmtfs drop-in `-P -r -v` (**never `-s`**: that stops the modem on exit, which is the TrustZone hang), sensors (persist `ro,noload`, served tree under /run, hexagonrpcd started by udev), BT/board.bin copied, kmsgcap, usb0 on networkd, NM for WiFi, timesyncd. - `pd-mapper`, `tqftpserv`, `hexagonrpc`: built natively in an aarch64 chroot. `qrtr-git` and `rmtfs-git` come from ALARM's aur repo; libssc and iio-sensor-proxy from extra. - `gts6l-uboot` 2025.10-2: U-Boot with menu 38 (SD boot + serial console) and menu 39 (XBL+ABL log to SD). **Rootfs:** `alarm/out/gts6l-alarm-rootfs.ext4` (4 GB, 2.2 GB used), via `sudo WORK=<tmpfs> alarm/build-rootfs.sh extract && customize && image 4096`. Verified by mounting it: all packages present, stock kernel removed, every Aug 23 unit enabled, `/boot` has our kernel + DTB, and `resolv.conf` / `nsswitch.conf` / `pacman.conf` are stock. **Deploy:** `tools/deploy-sd.sh /dev/sdX --write` partitions the card, writes that image, and builds `boot-gts6l.img` + CRC from the rootfs's own kernel with the golden cmdline (`tools/mkboot-gts6l.sh` refuses any cmdline missing `ufs_qcom.dbg_rate_a=1`). **Deliberately NOT restored:** - The patched rmtfs. Stock `-r` is enough (the modem never asked for EFS) and can't corrupt Samsung's NV. The patch is parked in `alarm/upstream/rmtfs`. **Still open, needs the tablet:** - [ ] Deploy the card and boot it with the serial jig attached. - [ ] Verify each service in turn. The Aug 22 note says enabling the modem stack at boot once wedged the device; that was before the recovery=disabled fix. - [ ] The audio modules-load list is a best guess (the original was lost). Trim to what's actually needed. - [ ] Flash the new U-Boot with `gts6l-flash-uboot` and test menus 38/39. - [ ] Backup habit (step 5). Bugs found along the way: - `build-rootfs.sh` copied `resolv.conf` through ALARM's absolute symlink **onto the host's** `/run/systemd/resolve/resolv.conf`. Fixed. - The old `logsave`/`mirsave` raw SD writes would land inside the new card's FAT. Fixed.