kit/linux-gts6l
1
0
Fork 0
Code Issues 18 Pull requests Projects Releases Packages Wiki Activity Actions

Modem: rflm_qlnk assertion on ~half of boots, and SSR recovery then wedges the AP #16

New issue
Open
opened 2026-08-22 19:33:36 +00:00 by kit · 4 comments
kit commented 2026-08-22 19:33:36 +00:00
Owner
Copy link

The single biggest source of instability. The modem asserts in its RF Link Manager on roughly every other boot, and the remoteproc recovery that follows takes the application processor down with it.

Signature

Every dead boot ends the same way, at ~27 s:

[27.9s] qcom_q6v5_pas 4080000.remoteproc: watchdog received:
        rflm_diag_error.cc:368:RFLM@qsf_hl_seq.c:119
        Assertion (rflm_qlnk_ls_retry_cnt < 2) failed
[27.9s] remoteproc remoteproc1: crash detected in modem: type watchdog
[27.9s] remoteproc remoteproc1: handling crash #1 in modem
[28.0s] remoteproc remoteproc1: recovering modem      <-- last useful line
[31.2s] regulator: Not disabling unused regulators

and then, a few seconds to ~20 s later, the AP dies. Two captured examples of what comes immediately before death:

boot A: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
        5-...0: (0 ticks this GP) idle=819c/1/0x4000000000000000 softirq=4233/4233 fqs=5079
        (detected by 1, t=21004 jiffies, g=9493, q=1580 ncpus=8)
boot B: sched: DL replenish lagged too much

The failure mode from outside is a wedge, not a panic: the USB controller keeps the gadget enumerated in hardware, but nothing in software responds — no ping, no ACM, no console output. Writes to the ACM endpoint return EAGAIN because the device side never drains it.

Rate

From the persistent /root/kmsg.log (the gts6l-kmsgcap service), across 37 boots:

  • 19 boots hit rflm_qlnk_ls_retry_cnt — 51%.
  • Boots that hit it die.
  • Boots that do not hit it are stable: one ran 443 s with all four remoteprocs running (slpi/modem/cdsp/adsp), WiFi listing 16 networks, and audio playing.

RFLM/qlnk is the RF Link Manager talking to the RF front-end over QLink; ls_retry_cnt < 2 is a link-status retry exhaustion. It fires before anything in userspace touches the modem.

Two separable bugs

  1. The assertion itself — firmware-side, likely RF front-end init that mainline does not set up (rails/clocks the modem expects). Intermittent, so possibly a race or a marginal supply.
  2. SSR recovery kills the AP — this is the one that matters. A crashed modem should cost us WiFi, not the whole system. Every dead boot stops at recovering modem, and the RCU stall shows a CPU taking 0 ticks in a grace period, i.e. wedged with no timer progress. Prime suspect is the sysmon/SSCTL notification path plus glink teardown blocking with interrupts off or holding a lock — sysmon blocks ~5 s per notification against an unresponsive peer, and there are four remoteprocs to notify.

Fixing (2) makes the device usable even while (1) remains.

Correcting an earlier conclusion

An earlier "definitive single-variable test" recorded the audio PD binding (qcom,protection-domain = "avs/audio", "msm/adsp/audio_pd") as the cause of this assertion. That was wrong. It compared one boot per arm against an event that fires ~51% of the time — a single sample of a coin flip on each side. Verified since:

  • The assertion occurs on baseline with the PD binding reverted (confirmed: no qcom,protection-domain in the DTS, working tree clean against HEAD).
  • With the PD binding restored, a boot came up modem=running rflm=0 afe_err=0, audio played audibly, modem healthy.

So the PD binding is neither implicated nor exonerated. Any future comparison here needs many boots per arm, never one.

Next steps

  1. Test whether disabling recovery keeps the AP alive: write disabled to /sys/kernel/debug/remoteproc/remoteprocN/recovery before the assertion window. If the system survives with a dead modem, the recovery path is confirmed as the killer. (gts6l-norecover.service installed on the rootfs to do this early.)
  2. If confirmed, instrument the recovery path — qcom_sysmon, qcom_glink_*, rproc_crash_handler_work — to find what blocks.
  3. Separately, chase the RF assertion: compare the modem's expected RF front-end supplies against what our DTS provides.

Notes for whoever picks this up

  • ramoops does not survive a reset on this device (DRAM decays; uncorrectable error in header ×10 every boot). The log channel that works is /root/kmsg.log on the SD rootfs.
  • U-Boot is not involved. Its console shows a clean handoff: image CRC-verified, Uncompressing Kernel Image to 80008000, FDT relocated to 0xbcbe4000 (~700 MB clear of the kernel), Starting kernel ....
The single biggest source of instability. The modem asserts in its RF Link Manager on roughly every other boot, and the remoteproc recovery that follows takes the **application processor** down with it. ## Signature Every dead boot ends the same way, at ~27 s: ``` [27.9s] qcom_q6v5_pas 4080000.remoteproc: watchdog received: rflm_diag_error.cc:368:RFLM@qsf_hl_seq.c:119 Assertion (rflm_qlnk_ls_retry_cnt < 2) failed [27.9s] remoteproc remoteproc1: crash detected in modem: type watchdog [27.9s] remoteproc remoteproc1: handling crash #1 in modem [28.0s] remoteproc remoteproc1: recovering modem <-- last useful line [31.2s] regulator: Not disabling unused regulators ``` and then, a few seconds to ~20 s later, the AP dies. Two captured examples of what comes immediately before death: ``` boot A: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: 5-...0: (0 ticks this GP) idle=819c/1/0x4000000000000000 softirq=4233/4233 fqs=5079 (detected by 1, t=21004 jiffies, g=9493, q=1580 ncpus=8) boot B: sched: DL replenish lagged too much ``` The failure mode from outside is a wedge, not a panic: the USB controller keeps the gadget enumerated in hardware, but nothing in software responds — no ping, no ACM, no console output. Writes to the ACM endpoint return `EAGAIN` because the device side never drains it. ## Rate From the persistent `/root/kmsg.log` (the `gts6l-kmsgcap` service), across 37 boots: - **19 boots hit `rflm_qlnk_ls_retry_cnt`** — 51%. - Boots that hit it die. - Boots that do not hit it are stable: one ran **443 s** with all four remoteprocs running (`slpi/modem/cdsp/adsp`), WiFi listing 16 networks, and audio playing. `RFLM`/`qlnk` is the RF Link Manager talking to the RF front-end over QLink; `ls_retry_cnt < 2` is a link-status retry exhaustion. It fires before anything in userspace touches the modem. ## Two separable bugs 1. **The assertion itself** — firmware-side, likely RF front-end init that mainline does not set up (rails/clocks the modem expects). Intermittent, so possibly a race or a marginal supply. 2. **SSR recovery kills the AP** — this is the one that matters. A crashed modem should cost us WiFi, not the whole system. Every dead boot stops at `recovering modem`, and the RCU stall shows a CPU taking *0 ticks in a grace period*, i.e. wedged with no timer progress. Prime suspect is the sysmon/SSCTL notification path plus glink teardown blocking with interrupts off or holding a lock — sysmon blocks ~5 s per notification against an unresponsive peer, and there are four remoteprocs to notify. Fixing (2) makes the device usable even while (1) remains. ## Correcting an earlier conclusion An earlier "definitive single-variable test" recorded the audio PD binding (`qcom,protection-domain = "avs/audio", "msm/adsp/audio_pd"`) as the cause of this assertion. **That was wrong.** It compared *one* boot per arm against an event that fires ~51% of the time — a single sample of a coin flip on each side. Verified since: - The assertion occurs on **baseline with the PD binding reverted** (confirmed: no `qcom,protection-domain` in the DTS, working tree clean against HEAD). - With the PD binding **restored**, a boot came up `modem=running rflm=0 afe_err=0`, audio played audibly, modem healthy. So the PD binding is neither implicated nor exonerated. **Any future comparison here needs many boots per arm**, never one. ## Next steps 1. Test whether disabling recovery keeps the AP alive: write `disabled` to `/sys/kernel/debug/remoteproc/remoteprocN/recovery` before the assertion window. If the system survives with a dead modem, the recovery path is confirmed as the killer. (`gts6l-norecover.service` installed on the rootfs to do this early.) 2. If confirmed, instrument the recovery path — `qcom_sysmon`, `qcom_glink_*`, `rproc_crash_handler_work` — to find what blocks. 3. Separately, chase the RF assertion: compare the modem's expected RF front-end supplies against what our DTS provides. ## Notes for whoever picks this up - `ramoops` does **not** survive a reset on this device (DRAM decays; `uncorrectable error in header` ×10 every boot). The log channel that works is `/root/kmsg.log` on the SD rootfs. - U-Boot is not involved. Its console shows a clean handoff: image CRC-verified, `Uncompressing Kernel Image to 80008000`, FDT relocated to `0xbcbe4000` (~700 MB clear of the kernel), `Starting kernel ...`.
kit commented 2026-08-23 11:44:34 +00:00
Author
Owner
Copy link

Recovery-path hypothesis: evidence against it

Ran the experiment. Disabling SSR recovery did not prevent the wedge.

SOAK CYCLE 1
  early: up=452s rflm=0 modem=running recov=enabled
  +60s:  up=517s rflm=0 modem=running    <-- SURVIVED
SOAK CYCLE 2
  early: up=26s  rflm=0 modem=running recov=disabled
  +60s:  DEAD

Cycle 2 confirms the knob was genuinely disabled at t=26 s, one second before the assertion window, and the device died anyway. With rproc->recovery_disabled set, rproc_crash_handler_work() skips rproc_trigger_recovery() entirely — so if the wedge still happens, the sysmon/glink teardown path is not what kills the AP.

Caveat, stated precisely: I could not retrieve /root/ssr-experiment.log or the kmsg tail afterwards to confirm the assertion actually fired on that boot — the device wedged before I could read it, and stayed wedged. rflm=0 at t=26 s is consistent with the assertion landing at ~27 s (just after the probe) but does not prove it did. So this is strong evidence against hypothesis (2), not a clean disproof. Re-confirm by reading the persistent log from a boot that is known to have asserted.

Cycle 1 is consistent with the established pattern: no assertion, stable for 517 s.

Wedge is a hang, not a panic

Worth recording: in the wedged state the device still answers ICMP but sshd is dead, and the USB device number does not change — i.e. it never reboots. panic=10 sysctl.kernel.panic_on_rcu_stall=1 therefore does not rescue it, because nothing panics. The RCU stall is reported and the machine simply stops making progress. This also explains why several "it auto-recovered" observations were really the device having been power-cycled.

Practical consequence: the auto-reboot safety net does not exist for this failure mode, so unattended soak testing stalls on the first wedge and needs a human.

Tooling installed on the rootfs

  • /usr/local/bin/gts6l-norecover — polls for the modem rproc, writes disabled to its debugfs recovery, appends to /root/ssr-experiment.log.
  • /etc/systemd/system/gts6l-modem.service.d/norecover.conf — ExecStartPre= hook. This is the mechanism that works.

Three earlier attempts failed and are recorded so they are not retried:

  1. Standalone unit with WantedBy=sysinit.target — systemctl enable did not stick (preset: disabled), unit never ran.
  2. Same unit with ConditionPathExists=/sys/kernel/debug/remoteproc — condition failed; the modem rproc registers at ~t=17 s, well after sysinit.
  3. udev rule SUBSYSTEM=="remoteproc", ATTR{name}=="modem" — never fired.

Hooking ExecStartPre onto the already-enabled gts6l-modem.service avoids the enable and ordering races entirely.

Where that leaves the diagnosis

Both halves of the original split are now in question. The assertion correlates strongly with death (19/37 boots assert; asserting boots die, non-asserting boots run for minutes), but the mechanism is not the recovery path. Next candidates: the crash-notification path that still runs with recovery disabled (rproc_crash_handler_work → rproc_report_crash and its blocking notifier chain), or something the modem does to shared state/interconnect at assertion time that the AP cannot survive regardless of what remoteproc does next.

## Recovery-path hypothesis: evidence against it Ran the experiment. **Disabling SSR recovery did not prevent the wedge.** ``` SOAK CYCLE 1 early: up=452s rflm=0 modem=running recov=enabled +60s: up=517s rflm=0 modem=running <-- SURVIVED SOAK CYCLE 2 early: up=26s rflm=0 modem=running recov=disabled +60s: DEAD ``` Cycle 2 confirms the knob was genuinely `disabled` at t=26 s, one second before the assertion window, and the device died anyway. With `rproc->recovery_disabled` set, `rproc_crash_handler_work()` skips `rproc_trigger_recovery()` entirely — so if the wedge still happens, the sysmon/glink teardown path is **not** what kills the AP. **Caveat, stated precisely:** I could not retrieve `/root/ssr-experiment.log` or the kmsg tail afterwards to confirm the assertion actually fired on that boot — the device wedged before I could read it, and stayed wedged. `rflm=0` at t=26 s is consistent with the assertion landing at ~27 s (just after the probe) but does not prove it did. So this is strong evidence against hypothesis (2), not a clean disproof. Re-confirm by reading the persistent log from a boot that is known to have asserted. Cycle 1 is consistent with the established pattern: no assertion, stable for 517 s. ## Wedge is a hang, not a panic Worth recording: in the wedged state the device **still answers ICMP** but sshd is dead, and the USB device number does not change — i.e. it never reboots. `panic=10 sysctl.kernel.panic_on_rcu_stall=1` therefore does **not** rescue it, because nothing panics. The RCU stall is reported and the machine simply stops making progress. This also explains why several "it auto-recovered" observations were really the device having been power-cycled. Practical consequence: the auto-reboot safety net does not exist for this failure mode, so unattended soak testing stalls on the first wedge and needs a human. ## Tooling installed on the rootfs - `/usr/local/bin/gts6l-norecover` — polls for the modem rproc, writes `disabled` to its debugfs `recovery`, appends to `/root/ssr-experiment.log`. - `/etc/systemd/system/gts6l-modem.service.d/norecover.conf` — `ExecStartPre=` hook. **This is the mechanism that works.** Three earlier attempts failed and are recorded so they are not retried: 1. Standalone unit with `WantedBy=sysinit.target` — `systemctl enable` did not stick (`preset: disabled`), unit never ran. 2. Same unit with `ConditionPathExists=/sys/kernel/debug/remoteproc` — condition failed; the modem rproc registers at ~t=17 s, well after sysinit. 3. udev rule `SUBSYSTEM=="remoteproc", ATTR{name}=="modem"` — never fired. Hooking `ExecStartPre` onto the already-enabled `gts6l-modem.service` avoids the enable and ordering races entirely. ## Where that leaves the diagnosis Both halves of the original split are now in question. The assertion correlates strongly with death (19/37 boots assert; asserting boots die, non-asserting boots run for minutes), but the mechanism is not the recovery path. Next candidates: the crash-notification path that still runs with recovery disabled (`rproc_crash_handler_work` → `rproc_report_crash` and its blocking notifier chain), or something the modem does to shared state/interconnect at assertion time that the AP cannot survive regardless of what remoteproc does next.
kit commented 2026-08-23 18:08:00 +00:00
Author
Owner
Copy link

It is a start-time race, and the NV/rmtfs lead is dead

Two solid results.

1. rmtfs is not involved

Ran rmtfs -P -r -v with logging, then started the modem. The verbose log recorded zero requests — the modem never asked rmtfs for anything, before or during RF init. So the EFS/NV path (and the fsg write-protected-LUN shadow patch) is not where the RF failure comes from. That hypothesis is closed.

2. The assertion tracks when the modem is started

modem start (uptime) result
~17.6 s assert
~17.7 s assert
~17.9 s assert
~18.9 s assert
402 s clean
417 s clean
2633 s clean

Started late, the modem comes up healthy and stays healthy. With it running from a late start, all three subsystems work at once:

modem = running,  rflm = 0
WiFi  = 14 networks
audio = card 0: S6 - Samsung Galaxy Tab S6

That is the first time modem, WiFi and audio have been up together on this device.

Uncomfortable interaction with the fts fix

The touchscreen fix (241a0784e5eb) took boot from 722 s to 25.6 s. That is a big win on its own, but it means the modem now always starts at ~18 s — i.e. always inside the failure window. Boots that used to succeed by accident (because the machine was crawling and the modem started at 400 s) no longer do. The fts fix did not cause the modem bug, but it does make it hit every boot instead of about half.

Workaround attempt: inconclusive

Added ExecStartPre=/bin/sleep 90 as a drop-in on gts6l-modem.service and rebooted. The device brought up USB networking but never started sshd (port 22 refused for 200 s), then dropped off entirely — so the test never produced a verdict. Unclear yet whether 90 s is simply too short, or whether blocking a Type=oneshot unit that multi-user.target wants for 90 s interferes with the boot. Retry with the delay moved off the critical path (a separate timer unit, or a background ( sleep N; echo start > .../state ) &) rather than an ExecStartPre.

What to look for next

Something that is not ready ~18 s in but is by ~400 s. rmtfs is ruled out. Candidates worth checking against a boot log:

  • the other three remoteprocs (slpi/adsp/cdsp start ~13-14 s, only ~5 s before the modem) being fully initialised rather than merely running
  • sync_state() completion — gcc/gpucc/rpmhpd all report sync_state() pending well past the failure window
  • RPMh/RSC state left behind by U-Boot's own regulator votes (see the tcs_tx_done WARN note in MAINLINE-NOTES)

A bisect of the delay (30 / 45 / 60 / 90 s) would bracket the threshold and point at whichever of these completes there.

## It is a start-time race, and the NV/rmtfs lead is dead Two solid results. ### 1. rmtfs is not involved Ran `rmtfs -P -r -v` with logging, then started the modem. **The verbose log recorded zero requests** — the modem never asked rmtfs for anything, before or during RF init. So the EFS/NV path (and the `fsg` write-protected-LUN shadow patch) is not where the RF failure comes from. That hypothesis is closed. ### 2. The assertion tracks *when* the modem is started | modem start (uptime) | result | |---|---| | ~17.6 s | assert | | ~17.7 s | assert | | ~17.9 s | assert | | ~18.9 s | assert | | 402 s | clean | | 417 s | clean | | 2633 s | clean | Started late, the modem comes up healthy and **stays** healthy. With it running from a late start, all three subsystems work at once: ``` modem = running, rflm = 0 WiFi = 14 networks audio = card 0: S6 - Samsung Galaxy Tab S6 ``` That is the first time modem, WiFi and audio have been up together on this device. ### Uncomfortable interaction with the fts fix The touchscreen fix (`241a0784e5eb`) took boot from 722 s to 25.6 s. That is a big win on its own, but it means the modem now *always* starts at ~18 s — i.e. always inside the failure window. Boots that used to succeed by accident (because the machine was crawling and the modem started at 400 s) no longer do. **The fts fix did not cause the modem bug, but it does make it hit every boot instead of about half.** ### Workaround attempt: inconclusive Added `ExecStartPre=/bin/sleep 90` as a drop-in on `gts6l-modem.service` and rebooted. The device brought up USB networking but never started sshd (port 22 refused for 200 s), then dropped off entirely — so the test never produced a verdict. Unclear yet whether 90 s is simply too short, or whether blocking a `Type=oneshot` unit that `multi-user.target` wants for 90 s interferes with the boot. Retry with the delay moved off the critical path (a separate timer unit, or a background `( sleep N; echo start > .../state ) &`) rather than an `ExecStartPre`. ### What to look for next Something that is not ready ~18 s in but is by ~400 s. rmtfs is ruled out. Candidates worth checking against a boot log: - the other three remoteprocs (slpi/adsp/cdsp start ~13-14 s, only ~5 s before the modem) being fully initialised rather than merely `running` - `sync_state()` completion — gcc/gpucc/rpmhpd all report `sync_state() pending` well past the failure window - RPMh/RSC state left behind by U-Boot's own regulator votes (see the `tcs_tx_done` WARN note in MAINLINE-NOTES) A bisect of the delay (30 / 45 / 60 / 90 s) would bracket the threshold and point at whichever of these completes there.
kit commented 2026-08-24 09:17:47 +00:00
Author
Owner
Copy link

Root cause of the wedge found, and the assertion isolated from it

Two separate bugs were tangled together here. One is solved; the other is now bounded and no longer fatal.

The wedge: qcom_scm_pas_shutdown() never returns

When the modem crashes, rproc_boot_recovery() → rproc_stop() → qcom_pas_stop() calls qcom_scm_pas_shutdown(), an SMC into TrustZone to tear down a wedged modem. That SMC does not return. The CPU is left in EL3, which is why it:

  • stops ticking (0 ticks this GP)
  • freezes its dynticks state — idle=948c/1/0x4000000000000000 is byte-identical at t=47 s, 120 s and 193 s
  • cannot be NMI'd — Sending NMI from CPU 4 to CPUs 1: produces no backtrace, ever, while NMIs to CPUs 2 and 4 report normally

No software state does that. Everything else was downstream: RCU can't complete a grace period, RTNL's holder can never run, and sshd/NetworkManager/cfg80211 pile up in D. Seven CPUs stay fine, which is why the kernel keeps printing, the panel keeps updating and ICMP keeps working — and why this kept getting misdiagnosed as a hang.

Fix: write disabled to the remoteproc's recovery attribute before starting it. Verified 2026-08-24: modem asserts at ~26 s exactly as before, and the system survives — 0 RCU stalls, 0 D-state processes, ip link returns instantly, adsp/cdsp/slpi all running. An identical boot was previously unusable.

Stock reaches the same end by another route: restart_level=SYSTEM on all nine subsystems, i.e. Samsung reboots rather than ever recovering in place.

Committed: b0c82aaf1dbe (adds qcom_q6v5_pas.auto_boot=0 so userspace can set the knob before start).

Why this took so long: two confounders, both now fixed

a3f17b0850f5 — missing PRNG. sm8150.dtsi describes no RNG, so the CRNG seeded from interrupt timing alone and crng init done landed ~133 s in. getrandom() blocks until then; udevd blew its 90 s timeout and was SIGKILLed and restarted up to 7 times, and nothing went ready behind it. dev-mmcblk0p2.device ranged from 9 s to 32 minutes between identical boots. Modem start time tracked boot speed, so the assertion looked random when it was tracking a missing DT node. Every single-boot comparison made before this — the audio PD binding, cpuidle.off, the 90 s delay — was sampling that noise.

241a0784e5eb — fts driver. regulator_get()/enable()/put() on every power call, releasing rails with a non-zero enable count and calling regulator_disable() on handles that never enabled anything. Every I2C transfer to the touch IC returned -110, GPI DMA piled up CH STOP timeouts, and boot crawled. 13 kernel warnings → 1; I2C timeouts, GPI DMA errors and SMMU faults all → 0; boot 722 s → 25.6 s.

The assertion itself: still unexplained, but heavily constrained

The modem asserts rflm_qlnk_ls_retry_cnt < 2 about 9 s after coming up, when started at ~16 s. It does not assert when started after ~90 s, and stock never asserts at all — despite starting it at 6.8 s, earlier than our failing case. So "early" is not the discriminator.

Eliminated, each with a clean single-variable test on fast deterministic boots:

hypothesis result
DSP ordering (stock is modem-first) disproven — modem started first at 16.95 s, still asserted
ath10k contending for the RF front end disproven — modprobe.blacklist=ath10k_snoc, still asserted
rmtfs / NV / EFS path disproven — rmtfs -v logged zero requests
coredump reading modem-owned memory disproven — coredump is already disabled
cpuidle deep states disproven — cpuidle.off=1 confirmed active, still stalled
a missing or mis-voltaged RF rail disproven — full rail-by-rail diff against stock, no RF discrepancy

The AP is idle for those 9 s (log is silent between 19 s and the assertion), so the modem is failing on hardware state it finds, not on anything we do concurrently.

Also found while diffing rails (9d3b38d1c50c)

  • pm8150_l17 — touch AVDD, capped at 3008 mV here vs 3300 mV downstream. The FTS controller was under-volted by ~444 mV and could never reach its specified voltage.
  • pm8150_l18 — missing entirely; downstream enables it at 912 mV for the SS PHY and DisplayPort. No consumer here yet (USB is high-speed-only for bring-up), so this changes nothing today.

Suggested next steps

  1. Make recovery=disabled the shipped default and package the bring-up unit.
  2. qcom_scm_pas_shutdown() hanging the calling CPU forever is arguably a mainline bug worth reporting upstream — at minimum it should be survivable.
  3. For the assertion: the remaining delta is what TZ/XBL hands the modem, or a modem-side precondition Android's earlier boot satisfies. Comparing the two subsys-pil-tz load paths is the obvious next angle.
## Root cause of the wedge found, and the assertion isolated from it Two separate bugs were tangled together here. One is solved; the other is now bounded and no longer fatal. ### The wedge: `qcom_scm_pas_shutdown()` never returns When the modem crashes, `rproc_boot_recovery()` → `rproc_stop()` → `qcom_pas_stop()` calls `qcom_scm_pas_shutdown()`, an SMC into TrustZone to tear down a wedged modem. **That SMC does not return.** The CPU is left in EL3, which is why it: - stops ticking (`0 ticks this GP`) - freezes its dynticks state — `idle=948c/1/0x4000000000000000` is byte-identical at t=47 s, 120 s and 193 s - **cannot be NMI'd** — `Sending NMI from CPU 4 to CPUs 1:` produces no backtrace, ever, while NMIs to CPUs 2 and 4 report normally No software state does that. Everything else was downstream: RCU can't complete a grace period, RTNL's holder can never run, and sshd/NetworkManager/cfg80211 pile up in `D`. Seven CPUs stay fine, which is why the kernel keeps printing, the panel keeps updating and ICMP keeps working — and why this kept getting misdiagnosed as a hang. **Fix:** write `disabled` to the remoteproc's `recovery` attribute *before* starting it. Verified 2026-08-24: modem asserts at ~26 s exactly as before, and the system survives — **0 RCU stalls, 0 D-state processes, `ip link` returns instantly, adsp/cdsp/slpi all running**. An identical boot was previously unusable. Stock reaches the same end by another route: `restart_level=SYSTEM` on all nine subsystems, i.e. Samsung reboots rather than ever recovering in place. Committed: `b0c82aaf1dbe` (adds `qcom_q6v5_pas.auto_boot=0` so userspace can set the knob before start). ### Why this took so long: two confounders, both now fixed **`a3f17b0850f5` — missing PRNG.** `sm8150.dtsi` describes no RNG, so the CRNG seeded from interrupt timing alone and `crng init done` landed ~133 s in. `getrandom()` blocks until then; udevd blew its 90 s timeout and was SIGKILLed and restarted up to **7 times**, and nothing went ready behind it. `dev-mmcblk0p2.device` ranged from **9 s to 32 minutes** between identical boots. Modem start time tracked boot speed, so the assertion looked random when it was tracking a missing DT node. Every single-boot comparison made before this — the audio PD binding, `cpuidle.off`, the 90 s delay — was sampling that noise. **`241a0784e5eb` — fts driver.** `regulator_get()`/`enable()`/`put()` on every power call, releasing rails with a non-zero enable count and calling `regulator_disable()` on handles that never enabled anything. Every I2C transfer to the touch IC returned `-110`, GPI DMA piled up `CH STOP` timeouts, and boot crawled. 13 kernel warnings → 1; I2C timeouts, GPI DMA errors and SMMU faults all → 0; boot 722 s → 25.6 s. ### The assertion itself: still unexplained, but heavily constrained The modem asserts `rflm_qlnk_ls_retry_cnt < 2` about 9 s after coming up, when started at ~16 s. It does not assert when started after ~90 s, and stock never asserts at all — despite starting it at **6.8 s**, *earlier* than our failing case. So "early" is not the discriminator. Eliminated, each with a clean single-variable test on fast deterministic boots: | hypothesis | result | |---|---| | DSP ordering (stock is modem-first) | **disproven** — modem started first at 16.95 s, still asserted | | ath10k contending for the RF front end | **disproven** — `modprobe.blacklist=ath10k_snoc`, still asserted | | rmtfs / NV / EFS path | **disproven** — `rmtfs -v` logged **zero** requests | | coredump reading modem-owned memory | **disproven** — `coredump` is already `disabled` | | cpuidle deep states | **disproven** — `cpuidle.off=1` confirmed active, still stalled | | a missing or mis-voltaged RF rail | **disproven** — full rail-by-rail diff against stock, no RF discrepancy | The AP is idle for those 9 s (log is silent between 19 s and the assertion), so the modem is failing on hardware state it finds, not on anything we do concurrently. ### Also found while diffing rails (`9d3b38d1c50c`) - **`pm8150_l17`** — touch AVDD, capped at 3008 mV here vs **3300 mV** downstream. The FTS controller was under-volted by ~444 mV and could never reach its specified voltage. - **`pm8150_l18`** — missing entirely; downstream enables it at 912 mV for the SS PHY and DisplayPort. No consumer here yet (USB is high-speed-only for bring-up), so this changes nothing today. ### Suggested next steps 1. Make `recovery=disabled` the shipped default and package the bring-up unit. 2. `qcom_scm_pas_shutdown()` hanging the calling CPU forever is arguably a mainline bug worth reporting upstream — at minimum it should be survivable. 3. For the assertion: the remaining delta is what TZ/XBL hands the modem, or a modem-side precondition Android's earlier boot satisfies. Comparing the two `subsys-pil-tz` load paths is the obvious next angle.
kit commented 2026-08-25 10:49:53 +00:00
Author
Owner
Copy link

2026-08-24/25: the assertion is start-time dependent; SSR diagnosis corrected

The wedge (part 2 of this issue) — corrected diagnosis

recovery=disabled written before starting the modem makes the crash survivable: modem still asserts, 0 RCU stalls, 0 D-state, ip link returns instantly, all other subsystems stay up. Needs qcom_q6v5_pas.auto_boot=0 (commit b0c82aaf1dbe) so nothing starts before the knob is set. Stock reaches the same end by another route — restart_level=SYSTEM on all nine subsystems, i.e. it reboots rather than ever recovering in place.

Correction to the earlier claim that qcom_scm_pas_shutdown() hangs. It does not, unconditionally. Stopping a healthy subsystem works fine — echo stop/echo start on slpi goes offline then running again. The 21 s that write takes is glink_ssr timing out on the dead modem edge, not TrustZone. Only stopping an already-asserted modem hangs, and the RCU stall window opens the instant recovering modem is printed (CPU 4 silent, rcu_preempt starved on CPU 1).

Operationally: start/stop are no-ops on a crashed rproc (stop → -EINVAL since state != RUNNING, start → 0 with rproc->power still held). The only restart knob is echo recover > recovery, which is the hanging path. So one modem start per boot, and never write recover.

The assertion is deterministic in start time

Unattended sweep, one trial per boot, ~20 trials. Full data in baselines/modem-start-time-sweep.md.

modem started at outcome
16.7 – 60.7 s asserts, 8.7–14.8 s later, 13/13
75.3 – 121.0 s runs, 7/7

Hard boundary between 60 s and 75 s. The earlier "~51% of boots" figure was start-time variance, not a coin flip — the two confounders behind it are fixed (missing PRNG in DT a3f17b0850f5, crng init 133 s → 5.3 s; and the fts regulator/IRQ misuse 241a0784e5eb, boot 722 s → 25.6 s).

What QLINK actually is

From strings in the modem firmware itself (modem.b18): QLINK is the high-speed SerDes between the modem and the WTR/SDR RF transceiver — rflm_qlnk_wtr_serdes_fix_8KV1_ag.c, RFLM_QLNK_GEAR_SEL_8p5Gbps/3Gbps/1p5Gbps, QLNK fails to instantiate SDR object, rflm_qlnk_start_link_ls vs ..._hs, QLNK is up running in LS mode.

LS = Low Speed, not "link setup" — bring-up runs the link at low speed first, then trains to HS. So rflm_qlnk_ls_retry_cnt < 2 means the modem could not establish the low-speed serial link to the transceiver, twice.

Eliminated, each on evidence

  • DSPs — nodsp arm (modem at 20 s, adsp/cdsp/slpi never started) still asserts, twice.
  • Support-daemon ordering — pd-mapper/rmtfs/tqftpserv are up at 15.8–16.0 s, 4.8 s before the modem starts, and the modem reaches tqftpserv over QMI at t+2.1 s in both arms.
  • EFS/NV — rmtfs -v logs nothing at all, including on boots where the modem survives. It is never asked.
  • AP load, CPU freq, CPU voltage, DDR freq, bus contention — 8 spinning CPUs + 2 DDR streams at 120 s: survives.
  • Storage I/O — mmc0 irq drops 295/s → 3.1/s exactly at the boundary, which looked decisive. Tested properly: heavy SD I/O neither breaks a late modem (2/2 survive) nor rescues an early one (2/2 assert). Coincidence. This also exposed that the earlier "load doesn't matter" test used CPU spinners and dd to /dev/null and tmpfs — no storage I/O at all.
  • Modem boot pacing — SD load stretches the modem's own boot from 1.6 s to 5.7 s and changes nothing either way.
  • Accumulated AP idle — continuous load from 18 s through the 120 s start: survives.
  • RF-hardware quiescence — a whole boot with the modem never started, followed by an early start next boot: still asserts. So it is time since this boot, not recovery time from a previous modem run. This also breaks the alternation confound (until this pair, every early trial happened to follow a late one).
  • Temperature — and backwards from the guess: the device cools over the window (~44 °C at 17 s → ~33 °C at 90 s). The working condition is colder.
  • RPMh traffic (apps_rsc irq 80.6/s vs 81.2/s across the boundary) and display traffic (43.0 vs 44.5/s) — flat.
  • Proxy CX/MSS votes — mainline mpss_resource_init already votes both at INT_MAX, released on handover. Downstream votes corner 385 and unvotes at 0.56 s.
  • RF clock buffers — stock's own clk_summary has rf_clk1/2/3 at enable_count 0; the AP never votes them.
  • Samsung vs CAF on the bring-up path — diffed samsung-src/kernel against LA.UM.8.1.r1-16600-sm8150.0. Samsung's entire delta in subsys-pil-tz.c / peripheral-loader.c / subsystem_restart.c is boot markers, panic("Modem booting fail !"), silent-SSR and sec_debug plumbing. Nothing touching clocks, regulators, votes or RF. Stock's modem bring-up is stock Qualcomm PIL.

Nothing happens at the boundary

State snapshots at 20 / 62 / 78 s (interconnect votes, genpd states, enabled clocks, regulators, interrupt counts, power supplies) show no structural difference across 62→78 — only counters advancing and momentary CPU idle states. And the full kernel + journal log for 30–119 s contains exactly one line: regulator: Not disabling unused regulators at 30.7 s. There is no event to find.

Fixed along the way: mcfg was never served

Separate real bug. The modem asks for its carrier config over tqftpserv and was refused all four files, because the stock modem vfat partition was never mounted. tqftpserv's translate_readonly() walks /sys/class/remoteproc/*/firmware and takes each dirname, so modem_pr has to sit beside the modem firmware, not at the top of /lib/firmware. Fixed in gts6l-alarm (firmware-modem.mount + link); rejects per boot 6 → 2, and the remaining two are the ota_firewall/ruleset pair that stock also refuses.

Does not affect the assertion — the modem requests mcfg at t+11.9 s, three seconds after it dies in the failing arm. Verified: early start with mcfg served still asserts, 2/2.

A surviving modem brings up WiFi

With the modem running, ath10k_snoc gets WLFW from it, loads firmware and creates wlan0, with all four remoteprocs up. Not functional yet — board_id 0xff, random MAC, chan info: invalid frequency 0, 0 BSS on scan. Separate issue.

Caveat on all of the above

Every one of these trials ran on a platform that hung on ~50% of boots (#17, only diagnosed afterwards). The 13-vs-7 split is far too clean to be an artifact, but the sweep is worth re-running now that the device boots reliably.

Next

  1. Bisect the 60–75 s boundary further, on the now-reliable platform.
  2. The modem's own F3 log would settle this, but DIAG is not reachable: the modem does not advertise or ACK DIAG, DIAG_CTRL, DIAG_DATA or DIAG_CMD (the names downstream's diagfwd_rpmsg.c uses for edge "mpss") — all time out in qcom_glink_create_local(). Its advertised set is only DATA1-11, DS, IPCRTR, LOOPBACK_CTL_MPSS, SSM_RTR_MODEM_APPS, apr_apps2, apr_voice_svc, glink_ssr.
  3. Remaining route to the modem's internals: patch rproc_crash_handler_work to take the minidump when recovery is disabled (mpss has minidump_id = 3). Carries XPU-fault risk on the MSA-protected carveout.
  4. Report the crashed-modem pas_shutdown hang upstream.
## 2026-08-24/25: the assertion is start-time dependent; SSR diagnosis corrected ### The wedge (part 2 of this issue) — corrected diagnosis `recovery=disabled` written **before** starting the modem makes the crash survivable: modem still asserts, 0 RCU stalls, 0 D-state, `ip link` returns instantly, all other subsystems stay up. Needs `qcom_q6v5_pas.auto_boot=0` (commit `b0c82aaf1dbe`) so nothing starts before the knob is set. Stock reaches the same end by another route — `restart_level=SYSTEM` on all nine subsystems, i.e. it reboots rather than ever recovering in place. **Correction to the earlier claim that `qcom_scm_pas_shutdown()` hangs.** It does not, unconditionally. Stopping a *healthy* subsystem works fine — `echo stop`/`echo start` on slpi goes offline then running again. The 21 s that write takes is `glink_ssr` timing out on the dead modem edge, not TrustZone. Only stopping an **already-asserted** modem hangs, and the RCU stall window opens the instant `recovering modem` is printed (CPU 4 silent, `rcu_preempt` starved on CPU 1). Operationally: `start`/`stop` are no-ops on a crashed rproc (`stop` → `-EINVAL` since state != RUNNING, `start` → 0 with `rproc->power` still held). The only restart knob is `echo recover > recovery`, which is the hanging path. So **one modem start per boot**, and never write `recover`. ### The assertion is deterministic in start time Unattended sweep, one trial per boot, ~20 trials. Full data in `baselines/modem-start-time-sweep.md`. | modem started at | outcome | |---|---| | 16.7 – 60.7 s | **asserts**, 8.7–14.8 s later, 13/13 | | 75.3 – 121.0 s | **runs**, 7/7 | Hard boundary between **60 s and 75 s**. The earlier "~51% of boots" figure was start-time variance, not a coin flip — the two confounders behind it are fixed (missing PRNG in DT `a3f17b0850f5`, crng init 133 s → 5.3 s; and the fts regulator/IRQ misuse `241a0784e5eb`, boot 722 s → 25.6 s). ### What QLINK actually is From strings in the modem firmware itself (`modem.b18`): QLINK is the high-speed SerDes between the modem and the WTR/SDR **RF transceiver** — `rflm_qlnk_wtr_serdes_fix_8KV1_ag.c`, `RFLM_QLNK_GEAR_SEL_8p5Gbps/3Gbps/1p5Gbps`, `QLNK fails to instantiate SDR object`, `rflm_qlnk_start_link_ls` vs `..._hs`, `QLNK is up running in LS mode.` **LS = Low Speed**, not "link setup" — bring-up runs the link at low speed first, then trains to HS. So `rflm_qlnk_ls_retry_cnt < 2` means *the modem could not establish the low-speed serial link to the transceiver, twice*. ### Eliminated, each on evidence - **DSPs** — `nodsp` arm (modem at 20 s, adsp/cdsp/slpi never started) still asserts, twice. - **Support-daemon ordering** — pd-mapper/rmtfs/tqftpserv are up at 15.8–16.0 s, 4.8 s before the modem starts, and the modem reaches tqftpserv over QMI at t+2.1 s in *both* arms. - **EFS/NV** — `rmtfs -v` logs nothing at all, including on boots where the modem **survives**. It is never asked. - **AP load, CPU freq, CPU voltage, DDR freq, bus contention** — 8 spinning CPUs + 2 DDR streams at 120 s: survives. - **Storage I/O** — `mmc0` irq drops 295/s → 3.1/s exactly at the boundary, which looked decisive. Tested properly: heavy SD I/O neither breaks a late modem (2/2 survive) nor rescues an early one (2/2 assert). Coincidence. *This also exposed that the earlier "load doesn't matter" test used CPU spinners and dd to /dev/null and tmpfs — no storage I/O at all.* - **Modem boot pacing** — SD load stretches the modem's own boot from 1.6 s to 5.7 s and changes nothing either way. - **Accumulated AP idle** — continuous load from 18 s through the 120 s start: survives. - **RF-hardware quiescence** — a whole boot with the modem never started, followed by an early start next boot: still asserts. So it is time since *this* boot, not recovery time from a previous modem run. This also breaks the alternation confound (until this pair, every early trial happened to follow a late one). - **Temperature** — and backwards from the guess: the device *cools* over the window (~44 °C at 17 s → ~33 °C at 90 s). The working condition is colder. - **RPMh traffic** (`apps_rsc` irq 80.6/s vs 81.2/s across the boundary) and **display traffic** (43.0 vs 44.5/s) — flat. - **Proxy CX/MSS votes** — mainline `mpss_resource_init` already votes both at `INT_MAX`, released on handover. Downstream votes corner 385 and unvotes at 0.56 s. - **RF clock buffers** — stock's own clk_summary has rf_clk1/2/3 at enable_count 0; the AP never votes them. - **Samsung vs CAF on the bring-up path** — diffed `samsung-src/kernel` against `LA.UM.8.1.r1-16600-sm8150.0`. Samsung's entire delta in `subsys-pil-tz.c` / `peripheral-loader.c` / `subsystem_restart.c` is boot markers, `panic("Modem booting fail !")`, silent-SSR and sec_debug plumbing. Nothing touching clocks, regulators, votes or RF. Stock's modem bring-up is stock Qualcomm PIL. ### Nothing happens at the boundary State snapshots at 20 / 62 / 78 s (interconnect votes, genpd states, enabled clocks, regulators, interrupt counts, power supplies) show **no structural difference** across 62→78 — only counters advancing and momentary CPU idle states. And the full kernel + journal log for 30–119 s contains exactly one line: `regulator: Not disabling unused regulators` at 30.7 s. There is no event to find. ### Fixed along the way: mcfg was never served Separate real bug. The modem asks for its carrier config over tqftpserv and was refused all four files, because the stock `modem` vfat partition was never mounted. tqftpserv's `translate_readonly()` walks `/sys/class/remoteproc/*/firmware` and takes each dirname, so `modem_pr` has to sit *beside* the modem firmware, not at the top of `/lib/firmware`. Fixed in gts6l-alarm (`firmware-modem.mount` + link); rejects per boot 6 → 2, and the remaining two are the `ota_firewall/ruleset` pair that stock also refuses. Does **not** affect the assertion — the modem requests mcfg at t+11.9 s, three seconds *after* it dies in the failing arm. Verified: early start with mcfg served still asserts, 2/2. ### A surviving modem brings up WiFi With the modem running, `ath10k_snoc` gets WLFW from it, loads firmware and creates `wlan0`, with all four remoteprocs up. Not functional yet — `board_id 0xff`, random MAC, `chan info: invalid frequency 0`, 0 BSS on scan. Separate issue. ### Caveat on all of the above Every one of these trials ran on a platform that hung on ~50% of boots (#17, only diagnosed afterwards). The 13-vs-7 split is far too clean to be an artifact, but the sweep is worth re-running now that the device boots reliably. ### Next 1. Bisect the 60–75 s boundary further, on the now-reliable platform. 2. The modem's own F3 log would settle this, but DIAG is not reachable: the modem does not advertise or ACK `DIAG`, `DIAG_CTRL`, `DIAG_DATA` or `DIAG_CMD` (the names downstream's `diagfwd_rpmsg.c` uses for edge "mpss") — all time out in `qcom_glink_create_local()`. Its advertised set is only DATA1-11, DS, IPCRTR, LOOPBACK_CTL_MPSS, SSM_RTR_MODEM_APPS, apr_apps2, apr_voice_svc, glink_ssr. 3. Remaining route to the modem's internals: patch `rproc_crash_handler_work` to take the minidump when recovery is disabled (`mpss` has `minidump_id = 3`). Carries XPU-fault risk on the MSA-protected carveout. 4. Report the crashed-modem `pas_shutdown` hang upstream.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
gts6l/6.17
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kit/linux-gts6l#16
No description provided.