> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/evidence-and-seals/mrope-serving-path-rca-p7.5-p7.6.md).

# mRoPE Serving-Path RCA (P7.5 / P7.6)

*Evidence for* [***20 · mRoPE Serving-Path RCA***](/osintelligence/part-iv-the-evidence-what-worked/20-mrope-serving-path-rca.md)*. This is the investigation record behind the chapter: a production serving-path collapse first hypothesized as a Blackwell-CUDA defect, then **falsified by direct evidence** (a Windows crash log naming the wrong module), then correctly isolated to an M-RoPE heap over-read in llama.cpp core, fixed upstream and cherry-picked verbatim into the sovereign build. It also cleanly separates that crash class from a second, unrelated content-generation failure class the same runs surfaced.*

*Unlike the other pages in this section, this is **not a pre-registered experiment** and carries no canonical-prefix seal. It is a forensic evidence bundle: direct measurement, a Windows Application Event Log, and a cross-toolchain differential, graded A1 in the record. How the sealed experiments differ is on* [***Verifying a Seal***](/osintelligence/evidence-and-seals/verifying-a-seal.md)*.*

**Dates** 2026-04-14 (P7.5 falsification) → P7.6 (root cause + patch) · **Admiralty A1** (direct measurement + Windows Event Log) · upstream fix cherry-picked as sovereign patch `822047a0a`.

## Finding 1 — the hypothesis falsified (P7.5, negative result)

The motivating hypothesis was that a Blackwell **MMQ CUDA segfault** (the zenn.dev / NVIDIA-migration-guide / LM Studio 12.8 class) explained a serving-path collapse where, under sustained JSON-schema-constrained load, the child preset server died and every subsequent request timed out. A CUDA 12.8 recompile was built to test it. **The recompile did not resolve the collapse, and the Windows Application Event Log named why:**

```
Faulting application name: llama-server.exe
Faulting module name:      llama.dll
Exception code:            0xc0000005
Fault offset:              0x0000000000084c1a
```

The faulting module is **`llama.dll`, not `ggml-cuda.dll`** — the access violation is inside the core inference layer, upstream of the CUDA backend. A CUDA toolchain change cannot fix a bug that is not in the CUDA kernels. The Blackwell-MMQ hypothesis was disproven by construction, and the cross-toolchain differential made the falsification airtight, the failure signature was **byte-identical across two independently built toolchains**:

| Signature                            | CUDA 13.1 build   | CUDA 12.8 build (SHA `612db6188`)  |
| ------------------------------------ | ----------------- | ---------------------------------- |
| Faulting module (Windows Event 1000) | `llama.dll`       | **`llama.dll`** (identical)        |
| Exception code                       | `0xc0000005`      | **`0xc0000005`** (identical)       |
| Fault offset                         | `0x84c1a`         | **`0x84c1a`** (identical)          |
| Parent Router `/health`              | 200 (stays alive) | **200** (identical)                |
| Child preset server                  | terminates        | **terminates** (GPU falls to idle) |
| Clean rows before collapse           | 0–38              | 0–19 (slot/KV state drift only)    |

The record is candid about the blast radius: **every P6 CTI-NER A/B number pre-dating this finding was measured on a substrate with a latent serving-path defect** (valid for the clean-row window, suspect beyond \~20 sustained calls), which is precisely why the [CTI-NER evidence page](/osintelligence/evidence-and-seals/cti-ner-a-b-p6.md) is the *patched* v2 that supersedes the wedged v1. The negative result is called publishable in its own words: it converts a toolchain-pivot guess into a falsified hypothesis with conclusive evidence.

## Finding 2 — root cause and the upstream fix (P7.6)

The defect was isolated to an **M-RoPE heap over-read** in llama.cpp's `ubatch_reserve` path, reported and fixed upstream and cherry-picked verbatim into the sovereign build rather than hand-patched, the credit-by-name discipline the chapter foregrounds:

* [**llama.cpp issue #20093**](https://github.com/ggml-org/llama.cpp/issues/20093) — the heap over-read and its two-part fix (the `ubatch_reserve` resize + the `state_read_meta` M-RoPE broadcast), diagnosed and authored by the issue reporter, applied verbatim.
* [**llama.cpp issue #19915**](https://github.com/ggml-org/llama.cpp/issues/19915) (reporter: KernelFreeze) — a K-shift assertion, fixed by **Georgi Gerganov** and reviewed by **@ngxson** in [**PR #19928**](https://github.com/ggml-org/llama.cpp/pull/19928) (merged as commit [`99bd67c`](https://github.com/ggml-org/llama.cpp/commit/99bd67c9b29851f758c7d22caa8fc57fd5af3e4f)), cherry-picked verbatim.

The composed result is the sovereign serving binary `llama-server.exe` SHA-256 `5f689cb4…53a98` (llama.cpp b7992 + sovereign patch `822047a0a`), the build the patched CTI-NER A/B then ran on. Its production-scale validation is the receipt: **zero crash-class events across 2,004 served rows**. The patch bundle itself (`p76_rope_mrope_kvcache_fix.patch`) is retained in the evidence directory.

## Finding 3 — the content class, kept separate (P7.6 discipline)

The same patched run surfaced a second failure mode, and the record's discipline is that it is **categorically separated from the crash class** rather than conflated (the mistake the P7.5 era made when a single `ok=False` count mixed two unrelated modes):

| Attribute          | Crash class (P7.6-resolved)          | Content class (baseline-model characteristic) |
| ------------------ | ------------------------------------ | --------------------------------------------- |
| HTTP status        | 500 proxy-timeout (child dead)       | **200** (request completed)                   |
| Server process     | child terminates, run wedges         | **stays up, run continues**                   |
| Windows Event 1000 | one per crash                        | **none**                                      |
| Mechanism          | `llama.dll` `0x84c1a` heap over-read | `n_predict`-budget exhaustion under grammar   |

The content class is `n_predict`-exhaustion under grammar-constrained decoding: uniform \~35.5 s latency (1,024 tokens × \~35 ms), a truncated-JSON `JSONDecodeError`, 3 baseline rows + 1 candidate row out of 2,004. The record establishes it as a **documented, community-corroborated baseline-model class, not a substrate defect** (Alibaba's own Qwen docs warn that a `max_tokens` cap truncates constrained JSON; Ollama #14570 and Simon Willison's grammar note describe the exact mechanism), and reads it as *evidence for* the CTI-NER thesis, the in-distribution weakness the sovereign LoRA is built to correct, not against it. Keeping the two classes separate is the same evidentiary hygiene the seals enforce, applied to an RCA.

## Evidence bundle (retained on disk)

The P7.5 finding directory carries the primary forensic artifacts: `sidecar.jsonl` (36-row harness telemetry, 20 ok / 16 fail, first failure row 20), `winevent_app_utf8.txt` (the Event 1000 access-violation capture), `nvidia_smi.txt` (GPU idle at wedge, child gone), `proc_snapshot.json`, `health_probe.txt` / `inference_probe.txt` (health 200, inference timeout post-collapse), and `winevent_system.json`. The P7.6 directory carries `content_failures_analysis.md` (the crash-vs-content separation + community corroboration table), `web_research.md`, the `ab_run` telemetry, and `patches/p76_rope_mrope_kvcache_fix.patch`. This page reproduces the two findings' analysis faithfully; the raw bundle remains in the sealed repository as the working substrate.

***

*Evidence & seals · mRoPE Serving-Path RCA (P7.5 / P7.6) · a forensic evidence bundle (Admiralty A1), not a pre-registered experiment — no canonical seal; upstream credit and the cherry-picked fix are on the record · CC BY 4.0 · © Jamey Kistner, OSINTelligence LLC*