> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/part-iv-the-evidence-what-worked/20-mrope-serving-path-rca/the-quick-version.md). # The quick version **Continue the tour →** [Next: 21 · The Watcher, the quick version](/osintelligence/part-iv-the-evidence-what-worked/21-the-watcher/the-quick-version.md) The short version of Chapter 20, three ways: the video walks the argument in a few minutes, the deep dive talks it through at a listening pace, and the infographic holds the whole chapter in one view. The full root-cause analysis, with the cross-toolchain falsification, the upstream fix validated across 2,004 production rows, and the attribution-preserving patch bundle, lives in the chapter itself: [20 · mRoPE Serving-Path RCA](/osintelligence/part-iv-the-evidence-what-worked/20-mrope-serving-path-rca.md). {% embed url="" %} **The deep dive.** A podcast-style audio conversation about this chapter: two AI hosts walk through the argument, the incidents behind it, and what it means, at a listening pace. Generated in Google's Gemini LM (formerly NotebookLM) from the chapter itself; the link opens the audio on Google's site. {% embed url="\_" %} *The conversation is AI-generated: an interpretation of the chapter, not the chapter. It can compress, paraphrase, or get details wrong. The written chapter is the authoritative, canonical source:* [*20 · mRoPE Serving-Path RCA*](/osintelligence/part-iv-the-evidence-what-worked/20-mrope-serving-path-rca.md)*.* *** ![The mRoPE Serving-Path RCA, the chapter in one view.](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2FCJqZWvUIeS4qKRJ5LZ0P%2Fmrope-infographic.png?alt=media) *** ### Chapter notes Section-by-section notes in two registers: the technical note on the left, the same idea in plain language on the right. Every row is one idea, so you can read straight across from one register to the other. The technical terms stay visible in the plain column on purpose; they are the vocabulary worth keeping. #### Abstract **The point:** a production crash, a wrong hypothesis killed on purpose, the real bug found in the upstream tracker, and three borrowed fixes validated across 2,004 clean rows. | The technical note | In plain language | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A Windows 0xC0000005 access violation in llama.dll at offset 0x84c1a wedged the production A/B after 20–40 sustained requests; the initial Blackwell-MMQ CUDA-kernel hypothesis was tested by a deliberate cross-toolchain rebuild (13.1 → 12.8) that reproduced the identical signature, falsifying it and relocating the defect to architecture-neutral core inference. | The serving engine kept crashing after a few dozen requests (**the wedge**), and the obvious suspect, the GPU toolchain, was executed against on purpose: rebuild on the other toolchain and see if the crash survives. It survived (**the falsification**), which moved the search from the GPU code to the engine's core. | | Cross-referencing the upstream tracker surfaced issue #20093, a heap over-read in ubatch\_reserve on M-RoPE models, as the exact primary cause; three surgical patches from two upstream reports (+18/−2 lines across two files) were applied to the pinned tag and validated on the 2,004-row production A/B: zero access violations, zero Event 1000 entries, zero router wedges. | The real bug was already diagnosed in the open-source project's own issue tracker (**the heap over-read**): a memory-sizing mistake specific to this model family. Eighteen borrowed lines fixed it, and the run that used to die before row 40 completed all 2,004 rows clean (**the validation**). | | Six non-crash failures in the same run are separated as a categorically distinct, community-documented token-budget-exhaustion-under-grammar class, corroborated by six independent sources rather than folded into the crash-class win. | Six leftover failures are deliberately NOT claimed as fixed (**the separated class**): they are a different, documented phenomenon confirmed across six independent vendors, and the paper refuses to let the headline absorb them. | #### 1. Introduction **The point:** the field has a breadth catalogue of inference-engine bugs; this is the depth study, and the honest posture is assembler, not inventor. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Liu et al. (2025) catalogued 929 inference-engine bugs across five engines with Crash the dominant symptom (65%); what the record lacks is detailed case-study RCA of a specific production defect, especially one that survives a toolchain flip; single-operator sovereign deployments cannot absorb multi-week blind debug cycles. | Published research counts inference-engine bugs in bulk (**the breadth catalogue**); almost nobody publishes the complete story of chasing one bug to ground. For a solo operator, weeks of blind debugging is not survivable, which is why the method matters. | | Four contributions: the falsification-first RCA pattern, the depth case study of #20093 with its two-part reporter-authored fix plus the #19915 K-shift cherry-pick, the 2,004-row validation harness, and the attribution-preserving patch bundle offered as operating doctrine. | Four deliverables (**the contributions**): a reusable debugging method, the worked example, the proof it held under production volume, and a paperwork discipline for shipping other people's fixes with their names attached. | | The companion papers (Ch14, Ch15) assumed a stable serving substrate; this paper is that prerequisite: the infrastructure-as-first-class-science claim that the sovereign stack can diagnose and patch its own substrate defects without vendor escalation. | The placement in the series (**the prerequisite**): the two chapters before this one quietly assumed the serving engine works. This chapter is the receipt that when it broke, the one-person stack fixed it without calling anyone. | #### 2. Background **The point:** why this bug exists at all: a position-encoding innovation crossed an engine assumption, and the serving topology made the latent defect near-certain to fire. | The technical note | In plain language | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Classical RoPE carries one position scalar per token; Qwen2-VL's M-RoPE carries a (t, h, w) triple, and the Qwen 3/3.5 family generalizes further with MRoPE-Interleave; llama.cpp tracks this as n\_pos\_per\_embd (1 classical, >1 M-RoPE), and every preset in the sovereign roster is a Qwen 3.5 descendant. | The background in one line (**M-RoPE**): newer multimodal models track several position numbers per token where older models tracked one. The engine's allocator still assumed one. Every model on this stack uses the new scheme, so the mismatch was everywhere. | | Router Mode's three flags compose the trigger surface: --models-preset (five registered presets), --models-max 1 (LRU eviction under the 12 GiB budget, exercising slot teardown on every switch), and --slot-prompt-similarity 0.3 (prompt-cache restoration through state\_read\_meta, the function hosting the companion bug); sustained same-slot traffic maximizes entry frequency. | The crash was not bad luck (**the trigger surface**): the exact memory-saving serving configuration this stack runs, one model loaded at a time with cache restoration on, is what hammers the buggy code path. A simpler setup would almost never hit it. | | The defect maps cleanly onto the Liu et al. taxonomy (symptom Crash; root cause memory management / improper buffer sizing on a model-variant-specific path), and the depth study contributes the cross-toolchain falsification evidence a static report cannot. | The bug slots into the published taxonomy exactly (**the taxonomy fit**), and what this chapter adds is the kind of evidence a survey cannot: the controlled experiment that proved which subsystem was innocent. | #### 3–4. The Failure and the Falsification **The point:** the deepest water: three decisive telemetry attributes, and a rebuild executed as an experiment rather than a fix. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The signature: HTTP 500 with the parent router healthy throughout, the child preset process terminated, Event 1000 naming llama.dll (not ggml-cuda.dll) at offset 0x84c1a, deterministic per toolchain (13.1 at rows 39–40, 12.8 at rows 20–22); the faulting-module attribute was the single most decisive evidence. | Three clues did the work (**the signature**): the crash lived in the child process, the dying module was the engine's core library rather than its GPU backend, and the crash row was repeatable per build, the fingerprint of accumulating corruption rather than a random flake. | | The rebuild was executed as a falsification step, not a remediation attempt: a remediation that succeeds leaves the operator uncertain why; a falsification is definitive in both directions, and the negative outcome (identical failure on 12.8) is the stronger scientific result, relocating the defect in one move. | The load-bearing distinction (**falsification, not remediation**): the toolchain swap was designed so that either outcome answers a question. The crash surviving the swap was the better answer: it eliminated an entire subsystem in one experiment, which no amount of version-bumping until it stops could have done. | | The negative result was committed and published as its own artifact; CUDA 12.8 is retained on its three independent merits (NVIDIA guidance, driver alignment, benchmark evidence), with only the remediation hypothesis refuted. | The wrong guess got its own paper trail (**the published negative**), and the toolchain change stayed for its own good reasons. The hypothesis died; the useful side effects lived. | #### 5. Root Cause **The point:** three bugs, one family: an allocator sized for yesterday's models, a restoration path that fills only the first slot, and a shift that should decline instead of assert. | The technical note | In plain language | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Primary (#20093): ubatch\_reserve sizes the position vector at n\_tokens; M-RoPE readers access pos\[j·n\_tokens+i] for j up to n\_pos\_per\_embd−1 (4 on Qwen 3.5), walking past the buffer end; the corruption is latent until a later dereference raises the violation at an unrelated offset, the canonical heap-corruption smell; the fix is a one-line multiplication, reporter-authored, applied verbatim. | The main bug (**the allocator defect**): the engine reserved one position slot per token, and the new models read four, silently trampling neighboring memory whose damage only surfaces later, somewhere else. The fix multiplies the reservation by the right number. One line. | | Companion (same report): state\_read\_meta writes only the zeroth position on cache restoration, leaving M-RoPE's remaining slots uninitialized: silently-wrong rotations at best, the observed crash at worst; the fix broadcasts the saved position across all dimensions, semantically correct for text-only sequences, exactly Router Mode's case. | The second bug lived in the cache-restore path (**the broadcast fix**): restoring a saved session filled in one of the four position values and left the rest as garbage. The fix copies the one real value across all four, which is exactly right for text work. | | Secondary (#19915 / PR #19928, authored by Georgi Gerganov, merged in b≥8148): get\_can\_shift() returned true unconditionally, so the scheduler attempted K-shift compaction on M-RoPE models and hit the assertion; the cherry-picked guard makes M-RoPE caches decline the shift; §5.4 explains the identical cross-toolchain failure: all three fixes live in llama.dll, so recompiling the CUDA backend could not have moved the defect. | The third fix teaches the cache to say no to an operation the new models cannot do (**the K-shift guard**), borrowed from an already-merged upstream commit. And the mechanism note closes the loop (**why both toolchains failed**): every fix lives in the core library the toolchain swap never touched, which is precisely what the falsification predicted. | #### 6–7. Resolution and Validation **The point:** eighteen lines, an atomic swap with three rollback layers, 2,004 clean rows, and six failures honestly fenced out of the win. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The bundle: +18/−2 lines across two translation units, one sovereign commit on the pinned tag, exported as a portable patch artifact whose header carries the full upstream bibliography (issue #20093, issue #19915, PR #19928, commit 99bd67c9b, reporter handles); nothing recomposed or paraphrased. | The whole fix is eighteen lines (**the patch bundle**), shipped with a bibliography: every chunk names the person who wrote it and where. Nothing was reworded, so the provenance stays walkable forever. | | The rebuild touches 46 objects and 43 link steps with zero flag changes, producing a binary whose SHA-256 is committed; the cutover is an atomic NTFS swap with three rollback layers preserved (patched 12.8 production, unpatched 12.8 falsification control, unpatched 13.1 original substrate), so every claim in the series can be re-run on its original substrate. | The fixed binary is fingerprinted and swapped in atomically (**the three-layer rollback**), with both broken builds archived on purpose: one is the scientific control, the other the original environment, and either can be restored without rebuilding. | | Validation: all 2,004 rows complete with zero 0xC0000005 events, zero Event 1000 entries, zero router wedges, including the 20–40-row historical danger zone; the run spans the same prompt, token-length, and sampling distribution as any operator workload; the manifest is the primary empirical artifact. | The proof is volume (**2,004 clean rows**): the exact workload that used to kill the server inside forty requests ran two thousand with not one crash-class event, through the very row range where every prior run died. | | The six non-crash failures: HTTP 200, body truncated mid-JSON, uniform \~35.5 s latency matching the 1,024-token budget exactly, the mark of token-budget exhaustion under grammar; corroborated by six independent sources (Willison, Alibaba/Qwen docs, Ollama #14570, LM Studio #1773, llama.cpp #20345, and the Ch14 companion); a content-class, not a substrate-class, engineered around rather than patched; the 24 h soak is queued as a stretch target, not a precondition. | The six leftovers get the honest treatment (**the fenced class**): same fingerprint every time, matching a documented industry-wide behavior of constrained decoding on a tight token budget, confirmed across six unrelated projects. Wrong kind of bug, wrong fix path, so they are engineered around, not claimed. | #### 8. Discussion **The point:** the method generalized with its own counter-cases, the attribution doctrine as load-bearing practice, and the assembler's posture stated plainly. | The technical note | In plain language | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A good falsification step has three properties: orthogonal to the hypothesis (one axis changes), definitive in both directions, and recoverable; the paper also names when NOT to use it (bug already reproduces minimally, instrumentation cheaper than rebuild, swap gates other work); the sovereign advantage is substrate control: the method costs one rebuild script, no approvals. | The method comes with its own limits (**the three properties**): change one thing, learn something either way, and be able to undo it. The chapter also lists when the opposite order is right, and why a one-person stack can run this play more cleanly than a committee can. | | The attribution doctrine is argued as load-bearing, not courtesy: preserved provenance makes the eventual upstream rebase mechanical and lets external readers assign credit correctly in both directions; the closing posture note assigns the hard diagnostic work to the community by name and claims only the falsification, the composition, and the validation. | The paperwork discipline earns its keep twice (**attribution as practice**): it makes the future merge mechanical and keeps credit honest. And the chapter's own credit line is deliberately modest (**the assembler's role**): the community wrote the fixes; this operator proved which hypothesis was wrong, assembled the pieces, and validated the result. | | §8.3 elevates cross-vendor corroboration to a first-class evidence tier: six independent implementations exhibiting the same signature make it a property of the mechanism, not a local defect, and without that discipline the substrate would have accumulated complexity against a non-defect; §8.4 bounds the claim: single substrate, 2,004-row window (weekly-timescale regressions out of scope), pre-merge patches requiring a future rebase, and no full minidump backtrace (the upstream match was definitive enough not to justify the cost). | Checking other vendors before patching is promoted to method (**the corroboration tier**): if six unrelated tools show the same behavior, it is the mechanism, not your bug. And the limits are printed: one machine, a bounded window, patches that will need reconciling when upstream catches up, and one diagnostic shortcut taken knowingly. | #### 9–11. Related Work, Ethics, and Reproducibility **The point:** breadth catalogue meets depth study, a disclosure posture inverted for good reason, and a rebuild any operator can verify. | The technical note | In plain language | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The paper sits on orthogonal axes to Liu et al. (breadth catalogue vs end-to-end depth study) and inherits the multimodal-RoPE lineage (Su 2021 → Qwen2-VL 2024 → MRoPE-Interleave): the defect is the mundane consequence of an engine predating the extension, and the seam between model architecture and engine implementation is where the next innovation will also break. | Two framings, both kept (**the seam**): as software-engineering evidence, and as a preview: every future position-encoding invention will cross this same engine seam, and this is what the crossing looks like when it snags. | | Ethics: no security dimension (crash, not disclosure or escalation), the fix already public upstream, no embargo; the responsible posture is inverted: publish immediately and in full (patch bytes, rebuild recipe, manifest, binary SHA-256) because every operator running Qwen 3.5 under Router Mode is at risk and has no cheaper remediation path; the upstream consolidating PR is scoped as the natural next step. | The disclosure logic runs backward from the usual (**publish everything now**): there is nothing secret to protect and many operators exposed, so the fastest ethical act is handing them the complete fix kit. | | Reproducibility: the full bundle is committed and mirrored (spec, patch file, research log, failure analysis, validation bundle, rebuild script); on a matched substrate the patch applies cleanly and the incremental rebuild completes in under five minutes against the pinned tag and commit. | The whole thing re-runs from the box (**the five-minute rebuild**): pinned tag, pinned commit, pinned script, fingerprinted output. |