Flaky boot: bootm runs in place from the fastboot buffer and may overlap the decompressed kernel #4

New issue

Open

opened 2026-08-22 17:24:33 +00:00 by kit · 2 comments

kit commented 2026-08-22 17:24:33 +00:00

Owner

Copy link

Booting a kernel — both fastboot boot and sdboot — fails intermittently. The image transfers fine (Finished. Total time: 0.5s, and sdboot even CRC-verifies it), then nothing comes up. Retrying the identical image usually works. This has been happening since early in the project and cost a dozen retries in one session; it is now the single biggest drag on iteration.

Likely cause: load/decompress overlap

Both paths boot in place from the fastboot download buffer:

fastboot=fastboot -l $fastboot_addr_r usb 0
sdboot=... load mmc 0:1 ${fastboot_addr_r} boot-gts6l.img ... bootm ${fastboot_addr_r}

But our boot images are built with --base 0x80000000 --kernel_offset 0x8000, so bootm relocates and gunzips the kernel to 0x80008000. Our Image.gz is ~12.9 MB and the decompressed Image is substantially larger. If that decompression reaches $fastboot_addr_r, it overwrites the source image mid-decompress — which would fail intermittently and in a size-dependent way, and would get progressively worse as the kernel grows. That matches the symptom exactly.

Relevant config:

CONFIG_FASTBOOT_BUF_SIZE=0x10000000   # 256 MB
CONFIG_FASTBOOT_BUF_ADDR=0            # runtime, from $fastboot_addr_r
CONFIG_SYS_BOOTM_LEN=0x4000000        # 64 MB decompress ceiling

Diagnosis (needs the U-Boot console)

From the serial console (gts6l-reboot console, menu entry 17):

printenv fastboot_addr_r kernel_addr_r fdt_addr_r loadaddr scriptaddr
bdinfo

Then compare $fastboot_addr_r against 0x80008000 + <decompressed Image size>. If they overlap, that is the bug.

Also worth capturing a failing attempt on the console — bootm prints its relocation/decompress steps, so it should be obvious where it dies (or whether it dies silently after handing off).

Likely fix

Load and boot from different addresses rather than in place: keep the download/read buffer well clear of the kernel's load+decompress range, or point bootm at a copy. Bumping $fastboot_addr_r high (well above 0x80008000 + SYS_BOOTM_LEN) is the simple version.

Related: u-boot-gts6l#1 proposes loading the kernel directly from ext4, which changes these addresses anyway — worth fixing the layout as part of that rather than twice.

Note

Not a kernel bug: the same image (identical CRC, verified) has booted successfully and then failed on the next attempt with nothing changed.

Booting a kernel — both `fastboot boot` and `sdboot` — fails intermittently. The image transfers fine (`Finished. Total time: 0.5s`, and `sdboot` even CRC-verifies it), then nothing comes up. Retrying the identical image usually works. This has been happening since early in the project and cost a dozen retries in one session; it is now the single biggest drag on iteration. ## Likely cause: load/decompress overlap Both paths boot **in place from the fastboot download buffer**: ``` fastboot=fastboot -l $fastboot_addr_r usb 0 sdboot=... load mmc 0:1 ${fastboot_addr_r} boot-gts6l.img ... bootm ${fastboot_addr_r} ``` But our boot images are built with `--base 0x80000000 --kernel_offset 0x8000`, so `bootm` relocates and gunzips the kernel to **0x80008000**. Our `Image.gz` is ~12.9 MB and the decompressed `Image` is substantially larger. If that decompression reaches `$fastboot_addr_r`, it overwrites the source image mid-decompress — which would fail **intermittently and in a size-dependent way**, and would get progressively worse as the kernel grows. That matches the symptom exactly. Relevant config: ``` CONFIG_FASTBOOT_BUF_SIZE=0x10000000 # 256 MB CONFIG_FASTBOOT_BUF_ADDR=0 # runtime, from $fastboot_addr_r CONFIG_SYS_BOOTM_LEN=0x4000000 # 64 MB decompress ceiling ``` ## Diagnosis (needs the U-Boot console) From the serial console (`gts6l-reboot console`, menu entry 17): ``` printenv fastboot_addr_r kernel_addr_r fdt_addr_r loadaddr scriptaddr bdinfo ``` Then compare `$fastboot_addr_r` against `0x80008000 + <decompressed Image size>`. If they overlap, that is the bug. Also worth capturing a failing attempt on the console — `bootm` prints its relocation/decompress steps, so it should be obvious where it dies (or whether it dies silently after handing off). ## Likely fix Load and boot from **different** addresses rather than in place: keep the download/read buffer well clear of the kernel's load+decompress range, or point `bootm` at a copy. Bumping `$fastboot_addr_r` high (well above `0x80008000 + SYS_BOOTM_LEN`) is the simple version. Related: u-boot-gts6l#1 proposes loading the kernel directly from ext4, which changes these addresses anyway — worth fixing the layout as part of that rather than twice. ## Note Not a kernel bug: the same image (identical CRC, verified) has booted successfully and then failed on the next attempt with nothing changed.

kit commented 2026-08-22 18:35:36 +00:00

Author

Owner

Copy link

Root cause found — and it is not the fastboot buffer

My original hypothesis above (fastboot download buffer overlap) was wrong. The actual bug is in the boot image header offsets, and it is deterministic and measurable.

Dumping the header of a known image:

kernel_size : 12966809 (0xc5db99)  addr 0x80008000
tags_addr   : 0x81e00000
ramdisk_addr: 0x82000000

But kernel_size is the compressed size. What matters for placement is the arm64 Image header's image_size field — text+data+BSS, the runtime footprint:

image_size  : 32309248 bytes = 30.81 MiB
kernel occupies 0x80008000 .. 0x81ed8000
DTB     @ 0x81e00000 : CLOBBERED by 864 KiB
ramdisk @ 0x82000000 : safe (by only 1.1 MiB)

The device tree sits 864 KiB inside the kernel's own BSS. The arm64 kernel zeroes its BSS during early startup, so the DTB is destroyed before it is ever parsed.

--tags_offset 0x1e00000 is the stock Android default, sized for a ~10 MiB kernel. A mainline defconfig build is 30.8 MiB and runs straight through it. Headroom before the DTB is 29.97 MiB; we are at 30.81 MiB — we crossed that line recently, which is exactly why the flakiness got worse over time rather than being there from day one.

Why it is intermittent rather than a hard failure: whether a given boot survives depends on U-Boot's FDT relocation policy (fdt_high), i.e. whether it leaves the DTB at tags_addr or relocates it clear before handing off. Same image, different outcome, nothing else changed — which is precisely the symptom reported.

Note ramdisk_addr had only 1.1 MiB of margin. It is unused today (ramdisk_size = 0), but it would have broken the moment we added the mkinitcpio initramfs.

Fix (applied)

New layout, 2 MiB aligned, all clear of the first firmware carveout at 0x85e00000:

kernel  0x80008000 .. 0x81ed8000   30.8 MiB now, room to grow to 48 MiB
tags    0x83000000
dtb     0x83100000
ramdisk 0x83400000  .. 0x85e00000  = 42 MiB available for an initramfs

tools/mkbootimg.py defaults updated (tags 0x01E00000 -> 0x03000000, dtb 0x01F00000 -> 0x03100000, ramdisk 0x02000000 -> 0x03400000) and build-617.sh updated to match.

Guard against regression

mkbootimg.py now parses the arm64 image header (transparently through gzip — only the first 64 bytes are needed) and refuses to build an image whose kernel footprint overlaps the DTB, ramdisk, or tags address. Verified against the old offsets:

$ python3 tools/mkbootimg.py ... --tags_offset 0x1e00000 ...
    kernel footprint 32309248 bytes (30.8 MiB): 0x80008000..0x81ed8000
mkbootimg: kernel occupies 0x80008000..0x81ed8000 (30.8 MiB incl. BSS) and overlaps:
  tags_addr = 0x81e00000  (864 KiB inside the kernel)
Move the offset(s) above the kernel, or pass --allow-overlap.

It returns silently for non-arm64 payloads (U-Boot itself, the 4.14 kernel), so the other build scripts are unaffected until they actually have a conflict. --allow-overlap is available as an escape hatch.

Consequence for past debugging

Every image we have booted in this project carried this defect. Any boot that "just didn't come up" may have been DTB corruption rather than the change under test. Differential results where only the DTB content changed are still meaningful — placement was identical on both sides — but individual boot attempts were unreliable, and conclusions drawn from a single failed boot should be re-checked now that the layout is fixed.

## Root cause found — and it is not the fastboot buffer My original hypothesis above (fastboot download buffer overlap) was wrong. The actual bug is in the **boot image header offsets**, and it is deterministic and measurable. Dumping the header of a known image: ``` kernel_size : 12966809 (0xc5db99) addr 0x80008000 tags_addr : 0x81e00000 ramdisk_addr: 0x82000000 ``` But `kernel_size` is the *compressed* size. What matters for placement is the arm64 `Image` header's `image_size` field — text+data+**BSS**, the runtime footprint: ``` image_size : 32309248 bytes = 30.81 MiB kernel occupies 0x80008000 .. 0x81ed8000 DTB @ 0x81e00000 : CLOBBERED by 864 KiB ramdisk @ 0x82000000 : safe (by only 1.1 MiB) ``` **The device tree sits 864 KiB inside the kernel's own BSS.** The arm64 kernel zeroes its BSS during early startup, so the DTB is destroyed before it is ever parsed. `--tags_offset 0x1e00000` is the stock Android default, sized for a ~10 MiB kernel. A mainline defconfig build is 30.8 MiB and runs straight through it. Headroom before the DTB is 29.97 MiB; we are at 30.81 MiB — we crossed that line recently, which is exactly why the flakiness got worse over time rather than being there from day one. Why it is intermittent rather than a hard failure: whether a given boot survives depends on U-Boot's FDT relocation policy (`fdt_high`), i.e. whether it leaves the DTB at `tags_addr` or relocates it clear before handing off. Same image, different outcome, nothing else changed — which is precisely the symptom reported. Note `ramdisk_addr` had only 1.1 MiB of margin. It is unused today (`ramdisk_size = 0`), but it would have broken the moment we added the mkinitcpio initramfs. ## Fix (applied) New layout, 2 MiB aligned, all clear of the first firmware carveout at `0x85e00000`: ``` kernel 0x80008000 .. 0x81ed8000 30.8 MiB now, room to grow to 48 MiB tags 0x83000000 dtb 0x83100000 ramdisk 0x83400000 .. 0x85e00000 = 42 MiB available for an initramfs ``` `tools/mkbootimg.py` defaults updated (`tags 0x01E00000 -> 0x03000000`, `dtb 0x01F00000 -> 0x03100000`, `ramdisk 0x02000000 -> 0x03400000`) and `build-617.sh` updated to match. ## Guard against regression `mkbootimg.py` now parses the arm64 image header (transparently through gzip — only the first 64 bytes are needed) and refuses to build an image whose kernel footprint overlaps the DTB, ramdisk, or tags address. Verified against the old offsets: ``` $ python3 tools/mkbootimg.py ... --tags_offset 0x1e00000 ... kernel footprint 32309248 bytes (30.8 MiB): 0x80008000..0x81ed8000 mkbootimg: kernel occupies 0x80008000..0x81ed8000 (30.8 MiB incl. BSS) and overlaps: tags_addr = 0x81e00000 (864 KiB inside the kernel) Move the offset(s) above the kernel, or pass --allow-overlap. ``` It returns silently for non-arm64 payloads (U-Boot itself, the 4.14 kernel), so the other build scripts are unaffected until they actually have a conflict. `--allow-overlap` is available as an escape hatch. ## Consequence for past debugging Every image we have booted in this project carried this defect. Any boot that "just didn't come up" may have been DTB corruption rather than the change under test. Differential results where only the DTB *content* changed are still meaningful — placement was identical on both sides — but individual boot attempts were unreliable, and conclusions drawn from a single failed boot should be re-checked now that the layout is fixed.

kit commented 2026-08-22 18:37:47 +00:00

Author

Owner

Copy link

Correction — the previous comment's root cause is wrong

I checked the U-Boot source instead of assuming, and my "DTB is clobbered" conclusion does not survive it. Retracting it.

tags_addr is never consumed by U-Boot. Grepping every use in the tree:

boot/image-android.c:103:  data->tags_addr = hdr->tags_addr;   # stored
boot/image-android.c:772:  printf("%stags address: %x\n", ...) # printed

That is all. Nothing reads it back. tags_addr is an ATAGS-era field; U-Boot places the FDT itself via boot_relocate_fdt(), which (with fdt_high unset, as ours is) allocates from lmb inside the bootmap rather than at the address in the image header. So the arithmetic in my previous comment was correct but described a placement that never happens. The DTB was not at 0x81e00000.

Two further things that argue against my earlier theories:

1. bootm_load_os() has an explicit source/destination overlap check that prints
   ERROR: new format image overwritten - must RESET the board to recover.
   So the original fastboot-buffer theory would produce a visible error, not a silent failure.
2. lmb_reserve(images->os.load, load_end - images->os.load, LMB_NONE) runs after decompression, so the decompressed kernel is reserved before the FDT is placed.

What is actually still a real hazard

One genuine issue survives, and it is narrower than what I claimed. U-Boot reserves load_end - load — the decompressed file size (31,750,656). The kernel's true footprint is the arm64 header's image_size (32,309,248), which includes BSS:

U-Boot reserves : 0x80008000 .. 0x81e4fa00
kernel occupies : 0x80008000 .. 0x81ed8000
unreserved gap  : 0x81e4fa00 .. 0x81ed8000   = 545 KiB

Anything U-Boot places in that 545 KiB window — FDT or ramdisk — is considered free memory by the bootloader and is then zeroed by the kernel as BSS. In practice boot_relocate_fdt() usually allocates high in the bootmap, well clear of it, so this may never bite. It is a latent hazard, not a demonstrated cause.

Status of the flaky boot

Unknown. I do not have a root cause, and I should not have claimed one. The way to get it is the serial console during a failing attempt — bootm prints each stage, so a failure between "Loading Kernel Image" and the kernel's own output will localise it immediately. That still needs doing.

What was changed anyway

The offset changes (tags 0x03000000, dtb 0x03100000, ramdisk 0x03400000) and the mkbootimg.py overlap guard are kept, but as hygiene, not a fix: the header should not advertise addresses that sit inside the kernel, and the guard documents the BSS-vs-file-size distinction that tripped me up. The ramdisk move is the one with concrete future value — 0x82000000 had only 1.1 MiB of clearance, which matters once the mkinitcpio initramfs exists.

No claim that any of this makes boots more reliable. That remains open.

## Correction — the previous comment's root cause is wrong I checked the U-Boot source instead of assuming, and my "DTB is clobbered" conclusion does not survive it. Retracting it. **`tags_addr` is never consumed by U-Boot.** Grepping every use in the tree: ``` boot/image-android.c:103: data->tags_addr = hdr->tags_addr; # stored boot/image-android.c:772: printf("%stags address: %x\n", ...) # printed ``` That is all. Nothing reads it back. `tags_addr` is an ATAGS-era field; U-Boot places the FDT itself via `boot_relocate_fdt()`, which (with `fdt_high` unset, as ours is) allocates from `lmb` inside the bootmap rather than at the address in the image header. So the arithmetic in my previous comment was correct but described a placement that never happens. The DTB was not at `0x81e00000`. Two further things that argue against my earlier theories: 1. `bootm_load_os()` has an explicit source/destination overlap check that prints `ERROR: new format image overwritten - must RESET the board to recover`. So the original fastboot-buffer theory would produce a **visible error**, not a silent failure. 2. `lmb_reserve(images->os.load, load_end - images->os.load, LMB_NONE)` runs after decompression, so the decompressed kernel *is* reserved before the FDT is placed. ## What is actually still a real hazard One genuine issue survives, and it is narrower than what I claimed. U-Boot reserves `load_end - load` — the **decompressed file size** (31,750,656). The kernel's true footprint is the arm64 header's `image_size` (32,309,248), which includes BSS: ``` U-Boot reserves : 0x80008000 .. 0x81e4fa00 kernel occupies : 0x80008000 .. 0x81ed8000 unreserved gap : 0x81e4fa00 .. 0x81ed8000 = 545 KiB ``` Anything U-Boot places in that 545 KiB window — FDT or ramdisk — is considered free memory by the bootloader and is then zeroed by the kernel as BSS. In practice `boot_relocate_fdt()` usually allocates high in the bootmap, well clear of it, so this may never bite. It is a latent hazard, not a demonstrated cause. ## Status of the flaky boot **Unknown.** I do not have a root cause, and I should not have claimed one. The way to get it is the serial console during a failing attempt — `bootm` prints each stage, so a failure between "Loading Kernel Image" and the kernel's own output will localise it immediately. That still needs doing. ## What was changed anyway The offset changes (`tags 0x03000000`, `dtb 0x03100000`, `ramdisk 0x03400000`) and the `mkbootimg.py` overlap guard are kept, but as **hygiene, not a fix**: the header should not advertise addresses that sit inside the kernel, and the guard documents the BSS-vs-file-size distinction that tripped me up. The ramdisk move is the one with concrete future value — `0x82000000` had only 1.1 MiB of clearance, which matters once the mkinitcpio initramfs exists. No claim that any of this makes boots more reliable. That remains open.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

gts6l/uboot

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kit/u-boot-gts6l#4

No description provided.