> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/part-iv-the-evidence-what-worked/20-mrope-serving-path-rca.md).

# 20 · mRoPE Serving-Path RCA

**Cross‑Toolchain Falsification and Surgical Patch‑Pick Resolution of a Production M‑RoPE Heap Over‑Read in llama.cpp: A Sovereign‑Deployment RCA Case Study**

*Chapter 20 · Part IV: What Worked · Evidence-backed · v1.0.0 · scaffolded and sealed 2026-04-14*

**Author:** Jamey Kistner, OSINTelligence LLC

**Keywords:** reliability engineering · falsification-first RCA · heap over-read · M-RoPE · llama.cpp · Qwen 3.5 · Blackwell sm\_120 · CUDA 12.8 vs 13.1

> **A companion paper.** This report is the infrastructure-forensics leg of the series: the production-reliability prerequisite beneath *Sovereign In-Distribution Imatrix Calibration* (Chapter 15) and *Sovereign CTI-NER* (Chapter 14), both of which assume the stable serving substrate this paper delivers. Its method (a deliberate, pre-registered falsification step before instrumented debugging) is the same falsification-first discipline the series applies to its own hypotheses.

> **What is new here.** The contribution is a transferable RCA pattern in which a deliberate, pre-registered substrate swap is inserted as a falsification step before instrumented debugging: the CUDA toolchain was flipped 13.1 to 12.8 not to fix the crash but to test whether the CUDA backend was the defect at all, and the identical failure surviving the flip relocated the bug from the GPU kernels to architecture-neutral core inference in one move. What that isolation then found was a latent M-RoPE heap over-read that sits on every Qwen 3.5 checkpoint and is invisible on classical single-position RoPE, resolved by three surgically applied upstream fixes and validated at 2,004 clean production rows against a build that had wedged in under 40. The second, portable contribution is a discipline rather than a result: an attribution-preserving patch bundle in which every hunk carries the upstream issue, PR, commit, and reporter handle whose fix it implements, so a fix shipped ahead of upstream merge keeps its provenance walkable indefinitely.
>
> **Deepest water.** §4, the falsification step read as method (orthogonal to the hypothesis, definitive in both directions, recoverable via the three-layer rollback stack), which is why the toolchain flip settled the question a forward-version bump could not; §5.4, the mechanism account of why both toolchains failed identically (the fix lives in llama.dll, not ggml-cuda.dll, so recompiling the backend could not have moved it); and §7.2, the six non-crash failures held out as a separately-corroborated token-budget class rather than folded into the crash-class win.

### Abstract

We report a production-severity defect-resolution case study on llama.cpp tag b7992 serving Qwen 3.5 models under Router Mode with JSON-schema-constrained decoding on a Windows 11 / CUDA 12.8 / RTX 5070 (sm\_120 Blackwell) substrate. The failure, a Windows 0xC0000005 access violation in llama.dll at offset 0x84c1a after 20–40 sustained requests, was initially hypothesized as a Blackwell MMQ CUDA kernel defect. A deliberate cross-toolchain rebuild (CUDA 13.1 → CUDA 12.8) reproduced the identical failure signature, falsifying the CUDA-backend hypothesis and relocating the defect to architecture-neutral core inference code. Cross-referencing against the upstream issue tracker surfaced llama.cpp issue #20093, a heap over-read in ubatch\_reserve on Multi-section Rotary Position Embedding (M-RoPE) models, as the exact primary cause. That single upstream report specifies a two-part fix, the ubatch\_reserve resize and a companion state\_read\_meta broadcast, both authored by its reporter; a second, independent upstream report (issue #19915) accounts for a related K-shift assertion. We applied three surgical patches drawn from these two upstream reports (+18/−2 lines across two files), rebuilt in-place, and validated the resolution against a 2,004-row production-endpoint A/B harness: zero 0xC0000005 events, zero Windows Event 1000 entries for llama-server.exe, zero router wedges. Six non-crash-class failures observed in the same run are reported as a categorically distinct community-documented token-budget-exhaustion-under-grammar class, separated from the crash-class resolution. The contribution extends the taxonomy of inference-engine defects catalogued by Liu et al. (2025) with a concrete depth-study, and offers a transferable falsification-first RCA pattern for sovereign-deployment operators investigating architecture-neutral vs architecture-dependent failure modes.

### 1. Introduction

Inference engines are critical infrastructure for locally-deployed language models, and the first systematic study of their defect landscape (Liu et al. 2025) catalogued 929 bugs across five engines with crashes as the dominant symptom class. What the published record still rarely contains is detailed case-study root-cause analysis of a specific production defect, especially one that survives a toolchain flip and thereby isolates architecture-neutral code as the culprit. Single-operator sovereign deployments cannot absorb multi-week blind debug cycles; the falsification-first RCA pattern this paper reports is the transferable methodology it offers.

#### 1.1 Contributions

1. A **falsification-first RCA pattern** that uses a deliberate toolchain flip as the isolation step, separating architecture-dependent (CUDA kernel) from architecture-neutral (core inference) defects before instrumented reproduction begins.
2. A **depth case study** of llama.cpp issue #20093 (M-RoPE heap over-read) on Qwen 3.5, including its two-part reporter-authored fix cherry-picked verbatim (the ubatch\_reserve resize and the companion state\_read\_meta M-RoPE broadcast, both specified in the #20093 report), plus a cherry-pick of separate issue #19915's K-shift guard (merged as PR #19928), together resolving the crash at b7992 without a forward version bump.
3. A **2,004-row production-endpoint validation harness** demonstrating zero crash-class events post-patch, with a categorically distinct non-crash-class failure mode (token-budget exhaustion under grammar) cleanly separated and corroborated against six independent community sources.
4. An **attribution-preserving patch bundle** format in which every patch hunk cites the upstream issue, PR, or reporter whose proposed fix it implements, offered as operating doctrine for sovereign operators contributing surgical fixes without a direct upstream merge path.

#### 1.2 Relation to companion papers

The imatrix companion (Chapter 15) established sovereign in-distribution calibration as a supply-chain-integrity property; the CTI-NER companion (Chapter 14) extended sovereignty into supervised fine-tuning. Both papers assumed a stable serving substrate. This paper is the production-reliability prerequisite: the infrastructure-as-first-class-science claim that the sovereign stack can diagnose and patch its own substrate defects without vendor escalation. The forthcoming monograph carries this paper as the "infrastructure rigor" leg of its evidence pyramid.

### 2. Background

#### 2.1 Rotary position embedding variants

Rotary position embedding (RoPE) in its original formulation (Su et al. 2021) encodes a single one-dimensional position per token via a rotation in the query/key spaces. A token at position *p* is rotated by an angle proportional to *p*, so the inner product between queries and keys becomes a function of relative position alone. Implementation cost is one position scalar per token.

Multimodal models broke this assumption. Qwen2-VL (Wang et al. 2024) introduced **three-dimensional M-RoPE**, in which a token carries a (*t*, *h*, *w*) triple (temporal, height, and width indices) rotated along three independent frequency bands. A text token uses *t* only; a video patch uses all three. The downstream consequence for any serving engine is that each token requires *three* position values, not one.

The Qwen 3 / 3.5 / 3-VL family further generalized this with **MRoPE-Interleave**, which removes the spatial-reset behavior and interleaves the frequency bands differently; the ICLR 2026 multimodal-positional-encoding survey covers the resulting design space and the Qwen3 technical report describes the specific adoption in the Qwen 3.5 roster. Every preset in the sovereign Router Mode stack (router-visual, orchestrator, architect, research-brain, ioc-classifier) is a Qwen 3.5 descendant and therefore exposes the same M-RoPE position-vector shape.

The llama.cpp implementation carries an integer property on each model descriptor, n\_pos\_per\_embd, that is 1 for classical RoPE and > 1 for any M-RoPE variant. This single scalar is the structural precondition for upstream issue #20093: the buggy allocator assumed the classical single-position case, so the defect lies latent on every Qwen 3.5 checkpoint but is invisible on single-dimensional RoPE models. It is a defect that sits at the intersection of model architecture and engine implementation, and it could only become visible once Qwen 3.5 M-RoPE variants entered the production serving path.

#### 2.2 llama.cpp Router Mode and the sustained-request trigger surface

llama.cpp's Router Mode is a multi-preset server configuration in which a single llama-server process hosts several registered model presets and switches between them on demand. Three flags govern its behaviour, each contributing a distinct part of the surface on which the bug manifests.

**--models-preset** registers one preset per INI section, each with its own model path, context size, and sampler defaults. The sovereign deployment registers five presets (four generalist + one LoRA). A chat-completions request targets a preset by name, which triggers the second flag.

**--models-max 1** bounds the number of simultaneously loaded presets. With 1, an incoming request whose preset is not currently loaded evicts the resident preset (LRU) and materialises the requested one from disk. This is the sovereign memory-budget choice that keeps 12 GiB of RTX 5070 VRAM feasible against 30 GiB of cumulative preset weight. Each eviction event invokes the slot teardown and reinitialization path, which in turn exercises the M-RoPE broadcast buffer paths on reload.

**--slot-prompt-similarity 0.3** during eviction and subsequent reload, llama.cpp attempts to preserve KV-cache state across compatible prompt prefixes by serializing and restoring slot state. The restoration invokes state\_read\_meta, the function hosting the companion M-RoPE bug this paper identifies: when the slot was written under M-RoPE but the read path assumes the classical single-position shape, the second and third position indices are left uninitialized. A sustained same-slot workload (the precise traffic shape of a 1,002-row evaluation harness) maximizes the frequency at which this path is entered.

Together the three flags compose a surface on which the latent M-RoPE defect is near-certain to trigger within O(10) requests. A simpler single-model, single-request configuration would almost never reach it; the bug is not a probabilistic flake but a structural consequence of the sovereign serving topology. This matters for the generalizability claim in §8: any operator running Qwen 3.5 presets under Router Mode with prompt-cache restoration enabled is at risk, and the patch bundle this paper describes is the minimal fix against the pinned upstream tag.

#### 2.3 Prior work on inference-engine bug catalogues

The only systematic empirical study of LLM inference-engine defects to date is Liu et al. (2025), which catalogued 929 bugs across five engines (llama.cpp, vLLM, DeepSpeed, MLC-LLM, TensorRT-LLM) and developed a taxonomy of 28 root causes against 6 symptom classes. Crash is the most prevalent symptom (65%); the llama.cpp subset comprises 124 reports. Our defect maps cleanly onto their taxonomy (symptom = Crash; root cause = memory management / improper buffer sizing on model-variant-specific code path), and the present paper contributes a full end-to-end depth study of one entry of that kind with cross-toolchain falsification evidence that a static-report analysis cannot produce.

### 3. Observed Failure

The production A/B harness (1,002 rows per arm, Qwen 3.5 family under Router Mode with JSON-schema-constrained decoding) collapsed at row ≈ 39 on CUDA 13.1 and at row ≈ 20 on CUDA 12.8. The per-event signature was identical across both toolchains:

| Attribute                      | Value                                                                                                                          |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| HTTP status, first failing row | 500 (proxy timeout; child connection dropped mid-completion)                                                                   |
| HTTP status, subsequent rows   | 500 (router still accepts; child no longer serves)                                                                             |
| Router parent /health          | 200 throughout                                                                                                                 |
| Child preset process           | terminated                                                                                                                     |
| Windows Event 1000             | Faulting application llama-server.exe; faulting module **llama.dll** (not ggml-cuda.dll); offset 0x84c1a; exception 0xC0000005 |
| GPU telemetry post-collapse    | 1,215 MiB VRAM / 0% utilization / 28 °C, baseline idle; preset weights released                                                |
| Reproducibility                | Deterministic per toolchain: 13.1 fails at rows 39–40; 12.8 at rows 20–22                                                      |

***Table 1.** The failure signature, identical on both toolchains.*

![Figure 1](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2Frwwrt1eJNwwCiHf5oGRZ%2Fgb-ch20-fig1.png?alt=media)

***Figure 1.** The wedge timeline: deterministic collapse at row 39 (CUDA 13.1) and row 20 (CUDA 12.8 unpatched); 2,004 clean rows on the patched build.*

Three attributes of this signature drove the root-cause search. First, the parent router survived: the router is a separate process from the preset-hosting child, and the child's death did not propagate. This localized the defect to the child's address space (the core inference process loading the GGUF weights) and ruled out router-layer bugs as primary causes. Second, the faulting module was llama.dll, not ggml-cuda.dll. Core inference code lives in the former; the CUDA backend lives in the latter. This was the single most decisive piece of evidence the telemetry surface produced. Third, the failure was deterministic per toolchain, not random: a fixed row-index wedge on each build indicated accumulating state corruption, not a sporadic race.

Full raw evidence (stderr captures, Windows Event Viewer exports, GPU telemetry snapshots, the harness sidecar JSONL for both pre-patch runs) is archived in the repository's falsification-evidence bundle alongside its 13.1 predecessor.

### 4. Falsification Method

The initial hypothesis (informed by the telemetry surface showing a Blackwell GPU running CUDA 13.1 on a consumer sm\_120 part) was that the defect lived in the CUDA 13.1 Blackwell MMQ (matrix-multiply-quantized) kernels. Three independent public sources supported this reading at the time: the NVIDIA Blackwell migration guide recommending CUDA 12.8 as the near-term stable toolchain for sm\_120, a March-2026 community benchmark showing unrelated MMQ regressions on CUDA 13.1 against Blackwell, and LM Studio's build notes for Blackwell recommending 12.8. Under this hypothesis, the rational remediation was a CUDA toolchain downgrade to 12.8, a multi-hour rebuild but a clean fix path.

**The rebuild was executed as a falsification step, not as a remediation attempt.** The distinction is load-bearing. A remediation attempt that succeeds leaves the operator uncertain *why* it succeeded: it may have fixed the real defect, or it may have silently swapped one defect class for another that happens not to reproduce in the current test. A falsification step is designed to produce a definitive answer in either direction: if the signature disappears, the CUDA-kernel hypothesis is confirmed; if the signature survives, the CUDA-kernel hypothesis is refuted and the search relocates to architecture-neutral code. The latter outcome is the stronger scientific result (a negative that narrows the hypothesis space) and cannot be reliably obtained from a blind forward-version bump that tries a different tagged release of the same code.

The 12.8 rebuild reproduced the identical failure at row 20 (earlier than the 13.1 failure at row 39, consistent with accumulating heap corruption hitting different allocation boundaries under different compile-time layouts, but semantically the same defect). The Blackwell-MMQ hypothesis was falsified. The formal record of the negative result is archived as its own finding; the negative result was committed to memory and published as a whitepaper-backing artifact in its own right, not treated as a silent failed remediation.

The falsification delivered two forward-moving signals simultaneously:

1. **Architecture-neutral code is the defect site.** Any further remediation must be a source-code patch in llama.dll, not a backend or kernel flip. The search space collapsed from "somewhere across the CUDA backend + core inference + build-flag surface" to "somewhere in core inference that M-RoPE-class models execute under sustained state-restoration load."
2. **CUDA 12.8 is retained as the Blackwell toolchain standard.** The 12.8 flip was justified by three independent considerations (NVIDIA guidance, Blackwell driver alignment, independent benchmark evidence) orthogonal to the specific failure, and those considerations remain valid after the falsification. The toolchain change is not reverted; only the remediation hypothesis is refuted.

This pattern (a **deliberate, pre-registered falsification step inserted into the debug plan before instrumented reproduction begins**) is the transferable methodological contribution of this paper. It is not a new technique in the software-engineering research literature, but in the narrower space of single-operator sovereign LLM deployments it is underused, with the more common alternative being forward-version-bumping until the crash stops (which conflates fix, masking, and defect swap).

### 5. Root Cause

#### 5.1 Primary: #20093 heap over-read in ubatch\_reserve

The primary defect is upstream issue [#20093](https://github.com/ggml-org/llama.cpp/issues/20093) in llama.cpp tag b7992, diagnosed and fixed by its reporter (credited in full under Upstream attribution). It sits in src/llama-batch.cpp, in the function llama\_batch\_allocr::ubatch\_reserve. The pre-patch allocator sizes the position vector for one position per token:

```cpp
// src/llama-batch.cpp, ubatch_reserve, pre-patch
udata->pos.resize((size_t) n_tokens);
```

Downstream readers in apply\_ubatch (same translation unit) access pos\[j \* n\_tokens + i] for j = 0 .. n\_pos\_per\_embd − 1. For a classical 1D RoPE model, n\_pos\_per\_embd = 1 and j is always 0, so the single allocation suffices. For any Qwen 3.5 M-RoPE variant, n\_pos\_per\_embd = 4 (three spatial dimensions plus one temporal), and reads at j ≥ 1 walk past the allocated buffer end. The over-read corrupts whatever follows the buffer on the heap; the corruption is latent (the process continues executing normally) until a subsequent operation dereferences the corrupted bytes and Windows raises the access violation reported via Event ID 1000. The fault is almost always reported at an offset that has nothing to do with the original over-read; this is the canonical heap-corruption diagnostic smell and a widely-documented debugging anti-pattern (Microsoft pageheap literature).

The fix is a one-line multiplication:

```cpp
// src/llama-batch.cpp, ubatch_reserve, post-patch (upstream #20093)
udata->pos.resize((size_t) n_tokens * n_pos_per_embd);
```

This patch is reporter-authored text, applied verbatim from the upstream issue thread; provenance is preserved in the patch file header.

![Figure 2](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2FaIeCVfHYhI0wpNKgManW%2Fgb-ch20-fig2.png?alt=media)

***Figure 2.** The allocator defect: one position slot per token allocated (classical RoPE assumption); M-RoPE readers walk n\_pos\_per\_embd strides past the buffer end.*

#### 5.2 Companion (same #20093 report): incomplete M-RoPE broadcast in state\_read\_meta

The companion defect is the second code path named in the same #20093 report; it lives in src/llama-kv-cache.cpp, in llama\_kv\_cache::state\_read\_meta. This function deserializes a previously-saved slot state during prompt-cache restoration (the code path invoked each time slot-similarity reuse elects to restore an evicted slot on a semantically-close incoming request). Sustained same-slot traffic turns this into a hot path.

Pre-patch, the restoration loop writes only the zeroth position index per token. For classical RoPE this is complete. For M-RoPE, the second through last position values per token stay uninitialized from prior slot contents. The subsequent apply\_ubatch call reads those uninitialized bytes and feeds them into the rotary embedding kernel, producing either a silently-wrong rotation (best case, drifts output quality) or a hard crash when the uninitialized values happen to be large and index out of later buffers (observed case in our signature).

The fix broadcasts the position to all M-RoPE slots on restoration:

```cpp
// src/llama-kv-cache.cpp, state_read_meta, post-patch
for (uint32_t j = 0; j < hparams.n_pos_per_embd(); ++j) {
    ubatch.pos[j * ubatch.n_tokens + i] = pos;
}
```

Like the resize, this broadcast is reporter-authored text applied verbatim from the same #20093 thread. Broadcasting the one saved position value across all M-RoPE dimensions is semantically correct for text-only sequences (all three spatial dimensions collapse to the temporal), which is the exact case Router Mode exercises. This patch is a companion to upstream #20093: the two bugs share a root cause (M-RoPE position buffer sizing) but manifest in two independent code paths. It is not assigned a separate upstream issue number at the time of writing; the fix was derived from the semantics of the batch-application code and the reporter's diagnosis in the #20093 thread.

#### 5.3 Secondary: K-shift guard missing for M-RoPE (PR #19928)

The third patch addresses a distinct but adjacent failure mode: the KV-cache K-shift path calling seq\_add() against an M-RoPE model, which is unsupported and triggers the assertion *"seq\_add() is only supported for n\_pos\_per\_embd() == 1"*, surfaced publicly as upstream issue [#19915](https://github.com/ggml-org/llama.cpp/issues/19915) (reported by KernelFreeze). Pre-patch at tag b7992, get\_can\_shift() returns true unconditionally, so the scheduler attempts the shift on Qwen 3.5 models and hits the assertion on the first K-cache compaction.

The upstream fix, authored by Georgi Gerganov (@ggerganov), landed as commit 99bd67c9b via [PR #19928](https://github.com/ggml-org/llama.cpp/pull/19928) and is merged in tags b ≥ 8148. Since our sovereign deployment is pinned to b7992 for other stability reasons (the two M-RoPE patches above are pre-merge), we cherry-pick the commit verbatim onto our tree:

```cpp
// src/llama-kv-cache.cpp, get_can_shift(), post-patch (PR #19928)
if (hparams.n_pos_per_embd() > 1) {
    return false;
}
return true;
```

The guard disables shifts for M-RoPE models; Qwen 3.5 KV caches now correctly decline K-shift compaction rather than asserting. Attribution is preserved in the patch file header: upstream issue, upstream PR, upstream commit SHA, and the author's GitHub handle.

#### 5.4 Why both CUDA 13.1 and 12.8 failed identically

The three patches above all live in src/llama-batch.cpp and src/llama-kv-cache.cpp (translation units compiled into llama.dll, core inference). None of them live in ggml-cuda.dll (the CUDA backend). This is why the toolchain flip did not eliminate the signature: the flip rebuilds the CUDA backend against a different nvcc, but the architecture-neutral core code compiles identically on both MSVC configurations. A heap over-read in architecture-neutral code cannot be fixed by changing which GPU compiler emitted the matrix-multiply kernels.

The row-20 (12.8) versus row-39 (13.1) delta in failure index has a secondary explanation consistent with this picture: the two builds produce subtly different heap layouts because of compile-time link ordering and inlining decisions, so the corrupted bytes land on different neighboring objects and the latent period between over-read and access-violation differs by a small integer multiple. The mechanism is identical; the onset timing is layout-sensitive, exactly as heap-corruption bugs behave under pageheap analysis.

The Blackwell-MMQ hypothesis that motivated the rebuild was a reasonable first guess. The publicly available telemetry surface emphasized the visible identifiers (*Blackwell*, *CUDA*, *sm\_120*) and three independent public signals pointed in the same direction. The falsification retains all three of those signals as still valid for their own narrower claims while removing them from this paper's hypothesis stack.

### 6. Resolution

#### 6.1 Patch bundle

The three patches compose to a +18 / −2 line delta across two translation units. The change is applied as a single sovereign commit 822047a0a on top of the pinned upstream tag b7992, preserved in-tree and exported as a portable patch artifact in the deep-debug bundle. The patch file header carries a bibliography of its three upstream provenances (issue #20093, issue #19915, PR #19928 with commit 99bd67c9b) together with the GitHub handle of each reporter or author. Nothing in the patch bundle is recomposed or paraphrased: each hunk is either verbatim from an upstream reporter's proposed fix text (patches 1 and 2) or a clean cherry-pick (patch 3). This is the "stand on shoulders" doctrine operationalized: the sovereign operator ships the fix locally ahead of upstream merge while preserving attribution indefinitely forward.

#### 6.2 Rebuild

The patched tree is rebuilt on the 12.8 toolchain (CUDA 12.8, MSVC 19.44, Ninja) via the pinned build script, which preserves the exact flag set used for the unpatched baseline. The incremental rebuild touches 46 object files and 43 link steps, producing a llama-server.exe whose SHA-256 is 5f689cb4…53a98. The hash is committed to the reproducibility bundle so that any future deployment can verify against it without re-running the build. No build flags changed between the unpatched and patched binaries (only the 18 source lines) which isolates the post-patch validation result (§7) to the patch bundle itself rather than to any toolchain or flag drift.

#### 6.3 Atomic swap and rollback layering

The sovereign cutover preserves three independent rollback layers so that any regression on the patched binary can be reverted without re-building:

1. **Patched 12.8** (current production): SHA-256 5f689cb4…53a98.
2. **Unpatched 12.8** (negative-result control): archived. This is the artifact that falsified the Blackwell-MMQ hypothesis; it remains on disk as the definitive comparator for "did the patch itself fix anything, or did some untracked environmental variable change?"
3. **Unpatched 13.1** (original substrate): archived, so that every experimental claim in the series can be re-run on its original substrate if required.

The swap pattern is atomic on NTFS: the replacement binary is staged alongside the live one, a single move under a narrow lock substitutes the new executable for the old, and a post-swap SHA-256 verification confirms the current binary matches the intended one. No intermediate "partial swap" state is externally observable. The rollback discipline means that the three validation runs reported in §7 are each reproducible from their original substrate, not merely from a current-state snapshot.

### 7. Validation

#### 7.1 Production-endpoint A/B: 2,004 rows, 0 crashes

The validation workload is the 1,002-row A/B harness from the CTI-NER companion paper: each row issues a chat-completions request to the sovereign Router Mode endpoint against the baseline and candidate presets, for a total of 2,004 request-response cycles against the patched binary. The pre-patch runs wedged the router at row 20 (12.8) and row 39 (13.1). The post-patch run completes all 2,004 rows with:

* **Zero** 0xC0000005 access-violation events. Windows Event Log shows no application-error entries attributable to llama-server.exe across the entire run.
* **Zero** router-wedge events. No HTTP 500 proxy-timeout signatures; no child-preset process terminations; the GPU-idle signature from the pre-patch runs never manifests.
* **Zero** failures in the 20–40-row historical danger zone on either variant, where every prior run had wedged.

![Figure 3](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2FCsPfkMvW7npRaKc3FeIo%2Fgb-ch20-fig3.png?alt=media)

***Figure 3.** Windows Event 1000 counts: present on both unpatched builds, absent across 2,004 post-patch rows.*

The absence of crash-class failures across 2,004 rows is the strongest signal available from a production-endpoint A/B: the 1,002-row dataset spans the same prompt distribution, token-length distribution, and sampling configuration as any operator-initiated workload. The patch bundle transforms a repeatable single-digit-row wedge into a sustained 2,004-row clean run with no other intervention. The validation manifest (row-by-row status, timing, and event-log correlation) is committed to the deep-debug bundle as the primary empirical artifact backing this paper.

#### 7.2 Non-crash-class failures (separated and corroborated)

The 2,004-row run produces six non-crash failures: three on the baseline variant and three on the candidate, including one row shared across both (prompt-specific, not substrate-specific). The failure signature on every one of these rows is identical and diagnostically distinct from the pre-patch wedge signature:

* HTTP 200 response (server healthy, response delivered)
* Response body truncated mid-JSON token, producing a parse error at the client step
* Uniform latency of approximately 35.5 seconds, matching the 1,024-token budget × per-token cost at the serving configuration exactly, the mark of token-budget exhaustion under grammar-constrained decoding rather than runtime failure

This class is publicly documented as a behavior of grammar-constrained decoding under GBNF / JSON-schema enforcement: when the grammar admits a continuation but the token budget expires mid-structure, the server emits the partial output without raising an error. Six independent corroborating sources describe the same class: Willison's grammar-constrained JSON analysis; Alibaba's Qwen structured-output documentation; Ollama issue #14570; LM Studio issue #1773; llama.cpp issue #20345; and the CTI-NER companion paper's §4.4.1, which observed the same class independently on the same substrate.

Because the mechanism is a documented community-confirmed property of constrained decoding (not a defect in the sovereign binary) the six rows do *not* count against the patch bundle's validation claim. They belong to a content-class, not a substrate-class. The operator's response to this class is to engineer around the documented limit (longer token budgets, grammar-aware budget allocation, streaming parse with partial-accept) rather than to treat it as an engine bug to be patched. The cross-vendor corroboration elevates the class from "anecdote" to "first-class documented behavior" for the purposes of this validation.

#### 7.3 Gate D (stretch): 24 h production soak

The 2,004-row A/B establishes absence of the crash-class failure across a request volume two orders of magnitude greater than the pre-patch wedge onset. A stretch validation target of a 24-hour continuous production soak under the natural sovereign workload (dossier generation, CTI NER extraction, router-visual image queries interleaved through the ordinary daily schedule) is queued as the Gate D dimension on the companion specification. The Gate D target is not a pre-condition for the claims in this paper, which rest on the controlled 2,004-row A/B and its absence-of-signature evidence. As of the scaffolding commit (2026-04-14), Gate D is queued but not started: the operator explicitly prioritized paper-quality doctrine over further empirical accumulation for this submission round.

### 8. Discussion

#### 8.1 Falsification-first RCA as transferable methodology

The transition from the falsified hypothesis to the source-verified fix generalizes to a method: when a failure signature points at a specific subsystem, execute a *deliberate, pre-registered substrate swap* designed to either confirm or eliminate that subsystem as the defect site, *before* committing to instrumented reproduction. A good falsification step has three properties: it is **orthogonal to the hypothesis** (changes exactly one axis, in our case the CUDA toolchain, while holding everything else fixed), **definitive in both directions** (a positive outcome confirms the hypothesis; a negative outcome eliminates it, rather than merely lowering its probability), and **recoverable** (the swap can be reversed without further damage to the production state, as the three-layer rollback stack in §6.3 ensures).

Falsification-first is not always the right approach. When the bug already reproduces in a minimal substrate, when instrumentation is cheaper than a rebuild, or when the substrate swap is itself expensive enough to gate other work for days, the sequence should invert: instrument, narrow, then swap only as a late-stage confirmation. The rebuild fit the pattern because three independent public sources agreed on the subsystem, the swap was a documented one-flag toolchain change, and the alternative (instrumenting CUDA kernel launches against a heap that was already corrupted upstream of the kernel) would have yielded misleading traces.

Single-operator sovereign deployments benefit disproportionately from this pattern because they carry full control of the substrate (the operator owns both the CUDA toolchain and the build pipeline) so the cost of a deliberate swap is bounded (a single rebuild script) rather than requiring coordination across vendor teams. In cloud-hosted or multi-team deployments the same method is theoretically available but practically costly: a substrate swap touches many stakeholders, and a swap-then-revert sequence may be politically expensive even when technically correct. The sovereign single-operator can execute the method as written, without approvals, and the outcome is reliably a falsifiable hypothesis rather than a blended-variable mystery. This is the methodological claim of the paper: *the single-operator substrate-control advantage translates into a scientifically stronger RCA path.*

#### 8.2 Attribution-preserving patch bundles

A patch bundle delivered ahead of upstream merge is an act of scientific debt: the operator inherits the fix but not the attribution trail. This bundle makes that debt explicit. Every hunk in the patch file carries a header block that names the upstream issue number, the GitHub handle of the reporter or author who proposed or wrote the fix, the upstream commit SHA (where one exists), and the upstream URL, so that any reviewer reading the patch six months or six years from now can walk the provenance graph without consulting the maintainer's memory. The sovereign commit message carries the same bibliography, ensuring the attribution trail survives both the patch-file representation and the git-history representation independently.

This is not a courtesy; it is a load-bearing scientific practice. Two failure modes make the case concrete. First, when upstream eventually merges the fix, the sovereign rebase must identify which hunks are equivalent to the upstream commit and which are independent; preserved provenance makes the diff reconciliation mechanical. Second, when an external reader attempts to reproduce our validation, they need to be able to distinguish our original empirical contribution (the 2,004-row A/B on sovereign hardware) from the upstream fix itself, otherwise they may attribute to us either too much or too little credit. The attribution header resolves both failure modes in advance.

The doctrine generalizes beyond this paper: any time a sovereign operator ships a local fix in advance of upstream merge, the patch should carry a bibliography and the commit should name the reporter. It is free to implement, cheap to maintain, and forward-compatible with any upstream-PR follow-up path the operator may eventually take.

A closing note on posture, since it shapes how this paper should be read. The load-bearing engineering here belongs to the community: the reporter of #20093 did the hard diagnostic work and wrote the fix; KernelFreeze surfaced the K-shift assertion; Georgi Gerganov and the llama.cpp reviewers resolved it upstream. This author's role was narrower and different in kind: to falsify a wrong hypothesis cleanly, to compose those already-authored pieces onto a pinned build, to preserve every attribution, and to validate the result on sovereign hardware. That assembler's role is the one this paper claims, and it is the way the author prefers to work: not around the open-source community but inside it, contributing the piece that was missing rather than reinventing the pieces that were not. The patch-bundle doctrine above is simply that preference written down as practice.

#### 8.3 Cross-vendor corroboration as a first-class evidence tier

The §7.2 token-budget-exhaustion class illustrates a second methodological principle: before treating any failure signature as a local defect, cross-reference it against every independent vendor implementation that should exhibit the same behavior. Six sources, each from a distinct organization (Willison, Alibaba Cloud / Qwen, Ollama, LM Studio, the llama.cpp upstream tracker, and our own companion paper on the same substrate) all describe the same mechanism under grammar-constrained decoding with bounded token budgets. When a signature matches across that many independent implementations, the probability that it is a general property of the mechanism rather than a local defect is overwhelming, and the sovereign operator's response correctly shifts from "patch substrate" to "engineer around the documented limit." Without this cross-vendor discipline, a patched substrate would have accumulated code complexity against a non-defect and obscured the signal of the crash-class failures that *were* substrate defects. The corroboration tier earns its status as a first-class category of evidence, co-equal with the primary academic literature and the upstream bug tracker.

#### 8.4 Limitations

The validation claim is bounded by four explicit limitations.

First, the evidence comes from **a single operator on a single substrate** (RTX 5070 sm\_120 Blackwell, Windows 11, CUDA 12.8, MSVC 19.44). The patch bundle is not validated on Linux, on non-Blackwell GPUs, on multi-slot configurations, or on deployments that exceed the 12 GiB VRAM envelope. The M-RoPE defect is architecture-neutral and should transfer, but "should" is not "does" until cross-substrate validation is performed. Other sovereign operators replicating the patch should report their own 2,004-row-class A/B outcomes as independent evidence.

Second, the outermost validation window is **the 2,004-row A/B, with a 24-hour soak queued as a stretch target**. Regressions on weekly or monthly timescales (slow memory fragmentation, driver-level resource leaks that only compound across days) are out of scope for this paper and must be observed in longer-duration production telemetry.

Third, **the patches are pre-merge upstream**. The M-RoPE fixes are cherry-picked from upstream issue-thread discussion; the K-shift guard is a cherry-pick from a merged upstream commit that post-dates our pinned tag. A future sovereign upstream-version bump to a tag that has merged any of these will require a rebase step, in which the cherry-picked hunks must be identified as equivalent to the merged upstream fix and dropped from the sovereign tree. The attribution-preserving patch header (§8.2) is deliberately designed to make this rebase mechanical.

Fourth, **the crash was not instrumented to a full Windows minidump backtrace**. The upstream issue #20093 match against the signature was definitive enough (the M-RoPE structural precondition, the llama.dll offset, and the reporter-diagnosed over-read pattern all agreed) that the cost of enabling full dump capture and repeating the wedge to obtain a symbol-resolved stack was not justified against the cost of simply applying the upstream-diagnosed fix and validating it. Future sovereign RCA cases where the upstream match is less definitive should plan the full analysis path as a first-line instrument.

### 9. Related Work

The primary anchor for the present paper is Liu et al. (2025), the first systematic empirical study of LLM inference-engine defects. Their contribution is breadth: a 929-bug corpus spanning five engines, a 28-root-cause taxonomy, a 6-class symptom distribution in which Crash dominates at 65%. Our contribution is complementary depth: a single entry from that catalog, followed end-to-end from telemetry signature through falsification through patch-level root cause through 2,004-row production-endpoint validation, with every artifact (patch file, rebuild script, validation manifest, negative-result bundle) committed and auditable. The breadth catalog establishes that inference-engine bugs are a first-class category of software defect worth systematic study; the depth study here establishes what a complete root-cause-to-validation record looks like for one such entry. Neither paper subsumes the other; they sit on orthogonal axes of the same research agenda.

The scientific lineage of the code path in which the #20093 defect lives is the multimodal-RoPE design space. Classical one-dimensional RoPE (Su et al. 2021) extended to three-dimensional M-RoPE in Qwen2-VL (Wang et al. 2024) and further generalized to the MRoPE-Interleave family surveyed at ICLR 2026. The defect itself is the mundane consequence of an engine implementation that pre-dates the M-RoPE extension: the allocator was correct for one-dimensional RoPE and silently incorrect for any positive-n\_pos\_per\_embd successor. The related-work framing is therefore dual: Liu et al. frames the paper as a contribution to empirical software engineering on inference engines, and the multimodal-RoPE lineage frames it as a contribution to understanding the model-architecture / engine-implementation seam that any new position-encoding innovation will traverse. Both framings should be retained.

### 10. Ethics and Responsible Disclosure

The defect identified in this paper has no security-sensitive dimension: it is a heap over-read in position-buffer sizing, not a memory-disclosure or remote-code-execution surface, and its realized impact in the wild is a process crash rather than information leak or privilege escalation. The fix itself was already public before this paper was drafted: upstream issue #20093 carries the diagnosis and the reporter's proposed patch text in plain view on the issue tracker, and PR #19928 / commit 99bd67c9b are merged upstream at the time of writing. There is no zero-day consideration and no embargo to honor.

The responsible disclosure posture for this paper is therefore the inverse of the usual: we publish the sovereign implementation immediately and in full, including the exact patch bytes, the rebuild recipe, the validation manifest, and the binary SHA-256, because every other operator running Qwen 3.5 presets under llama.cpp Router Mode with prompt-cache restoration is at risk of the same wedge and has no cheaper path to remediation than ours. Accelerating their path is a direct benefit to the open-weights ecosystem and carries no countervailing harm. A follow-up upstream pull request consolidating the three patches against a current-master base is the natural next step and is scoped for a future session; until that lands, the published patch-file artifact functions as a drop-in remediation any operator can apply against b7992.

Dual-use considerations: the falsification-first methodology generalizes to any software-engineering RCA and carries no AI-specific risk surface. The attribution-preserving patch-bundle doctrine is a net positive for both offensive and defensive ecosystems (it increases the transparency and auditability of the software supply chain in both directions) and is therefore dual-use-neutral.

### 11. Reproducibility Statement

The full reproducibility bundle is committed to the sovereign forge and mirrored publicly: the companion specification, the portable patch file, the web-research log, the content-class failure analysis, the A/B validation bundle, the companion A/B report, and the pinned rebuild script. Given access to a Windows 11 / RTX 5070-class substrate, CUDA 12.8 toolkit, MSVC 19.44, and llama.cpp upstream tag b7992 (commit 612db6188), the patch applies cleanly and the rebuild completes in under 5 minutes incrementally. The validation harness is the CTI-NER A/B suite against any in-distribution 1,000-row dataset.

***

*The Sovereign Stack · mRoPE Serving-Path RCA · Chapter 20 · Part IV · v1.0.0 · License CC BY 4.0 · © Jamey Kistner, OSINTelligence LLC*

**Citation (preferred):** Kistner, J. (2026). *Cross-Toolchain Falsification and Surgical Patch-Pick Resolution of a Production M-RoPE Heap Over-Read in llama.cpp: A Sovereign-Deployment RCA Case Study*, version 1.0.0. OSINTelligence LLC research whitepaper. Cited in-series by title.

*Upstream attribution, the reference list, and provenance follow as a sub-page of this chapter.*