> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/part-iv-the-evidence-what-worked/17-sovereign-big-model-compression.md).

# 17 · Sovereign Big-Model Compression

> **A companion paper.** This report is the second compression paper in the series and the large-scale successor to *Sovereign Domain Pruning* (Chapter 16), which established destructive expert-pruning on a 35B-class model. Where that paper pruned a model that already fit consumer VRAM, this one takes a 122B-parameter mixture-of-experts that does not, and shows the full arc: porting a published pruning method onto a model its authors never targeted, pruning under load without a datacenter, and serving the result at higher fidelity and equal speed than the unpruned model on the same 12 GB card. It shares the sovereign-corpus calibration thesis of *Sovereign In-Distribution Imatrix Calibration* (Chapter 15).
>
> **Status note.** This is a *living document*. The pruning, quantization, quality-gate, and vision phases (§4–§8) are sealed results with hash-anchored receipts; the on-box training and recipe-generalization phases (§9) are active work, reported here as pre-registered design with interim receipts and flagged as such. Every numerical claim traces to a sealed experiment or telemetry run named in the text; the reference list is audited in the companion ledger.

> **What is new here.** The contribution is the full arc of getting a 122B mixture-of-experts, shipped for the datacenter, onto a single 12 GB consumer card at production fidelity: porting a published one-shot expert-saliency method onto an architecture its authors never targeted (four documented adaptations, offered as reusable seams), pruning the model block-wise so the card never holds the whole network, and calibrating the prune on the operator's own workload so the experts removed are the ones that workload never routes to. The load-bearing result is the in-distribution asymmetry, measured on held-out data: removing a quarter of the expert weights costs about a third of the model's perplexity headroom on generic web text and essentially nothing (2.7%) on the sovereign workload it was pruned for, which is what lets the pruned 4-bit model replace the unpruned 2-bit build in production at two quantization tiers better fidelity and equal-or-better speed.
>
> **Deepest water.** §6 and Table 3, the perplexity split that is the whole thesis (+35% generic vs +2.7% sovereign), read with the honest register that the served gate is retention, not gain (every McNemar p between 0.75 and 1.00, so no delta is distinguishable from zero); §3, the four porting seams; and §8, the 35B replication reported in full as a composite FAIL, whose two honest negatives (a real memory-spill cliff where the dedicated-VRAM instrument lies, and a zero-replay corpus-slot mismatch caught by the mathematics gate) are the method's own guardrails, not asides.

### Abstract

The open-weight capability frontier is arriving as very large mixture-of-experts models whose total parameter count sits far above consumer memory even though only a fraction of experts activate on any token, and post-training quantization alone cannot close that gap for a 122B-class model on a 12 GB card, nor does it make the model trainable on-box. This paper takes the architecture-side route instead: one-shot expert pruning calibrated on the operator's own workload, which removes the experts that workload never routes to and shrinks the model toward the envelope in which it can be both served and, eventually, trained on a single consumer GPU. We report the full arc. We port a published one-shot expert-saliency method onto a 122B mixture-of-experts its authors never targeted, documenting the four adaptations that port required as reusable method. We prune the model block-wise so that a single 12 GB card never holds the whole network, under a synthesis-lock and two-axis hardware telemetry, to a text-only checkpoint. We calibrate an importance matrix on the sovereign corpus and quantize to low bit-width. We then subject the result to a pre-registered, hash-sealed served-quality gate against the unpruned model, and find that the pruned model at 4-bit quantization matches or beats the unpruned model at 2-bit on pooled mathematics, graduate-level knowledge, and a held math set, at equal-or-better throughput on the same card, so the pruned model replaced the unpruned one in production. We report, in full, a parallel 35B replication that returned a composite FAIL on a real memory-spill signature and forced a methodological reframe that the pre-registration made separable, and we describe the multi-token-prediction-head recovery and vision-projector enablement that followed. The on-box expert-offload training that motivates the whole arc is active work, reported as pre-registered design with interim receipts rather than as a result.

### 1. Introduction

#### 1.1 The problem

The most capable open-weight models now shipping are very large mixture-of-experts networks: a total parameter count in the hundreds of billions, of which only a handful of experts activate per token. That sparsity is what makes them fast to run, but it does not make them small to hold: the full expert bank must be resident, or paged, somewhere, and for a 122B-class model that requirement sits far above the memory of a consumer GPU. Post-training quantization narrows the gap by storing weights at low bit-width, but it bottoms out: below a certain precision quality collapses, and even at the floor a 122B model does not fit comfortably on a 12 GB card, where it must be served at an aggressive 2-bit setting that costs quality. Quantization also does nothing for the second problem, which is that the operator cannot train a model this size on-box at all: there is no headroom left for gradients. The gap this paper addresses is therefore twofold, fit and trainability, and quantization alone closes neither.

#### 1.2 The sovereign angle

The observation that makes the gap tractable is that a single operator does not need every expert. A mixture-of-experts model is trained so that its experts specialize across the entire pretraining distribution, but any one operator's actual workload exercises a stable subset of them and routes to the rest rarely or never. Those rarely-routed experts are not free: they occupy the memory that keeps the model off the card and consume the headroom that would otherwise allow training. In-distribution expert pruning removes exactly the experts the sovereign workload does not use, calibrated by observing the operator's own corpus rather than a generic proxy. This is the architecture-side mirror of the calibration thesis argued in *Sovereign In-Distribution Imatrix Calibration* (Chapter 15): there, the operator's corpus told the quantizer which weights to preserve in precision; here, it tells the pruner which experts to keep at all. The two compose, and this paper applies the quantization step downstream of the prune.

#### 1.3 Contributions

The paper makes seven contributions, each load-bearing and each tied to a sealed receipt or a flagged-active phase: (1) an engineering port of one-shot expert pruning onto a mixture-of-experts architecture the published method did not support, documented section by section as reusable method (§3); (2) a block-wise single-GPU pruning path that prunes a 122B model without the 12 GB card ever holding the whole network, under a synthesis-lock and two-axis hardware telemetry (§4); (3) an in-distribution importance-matrix calibration and low-bit quantization pipeline applied downstream of the prune (§5); (4) a pre-registered, hash-sealed served-quality gate showing the pruned 4-bit model matches or beats the unpruned 2-bit baseline at equal-or-better throughput on the same hardware (§6); (5) a fully-reported composite FAIL on a 35B replication and the methodological reframe it forced (§8); (6) multi-token-prediction-head recovery and vision-projector enablement on the pruned trunk (§7); and (7) the on-box expert-offload training design that motivates the arc, reported as active pre-registered work (§9).

### 2. Background and Prior Art

**Mixture-of-experts and activation sparsity.** A mixture-of-experts layer replaces a single feed-forward network with many, and a learned router sends each token to a small number of them, so that the parameters engaged per token are a fraction of the total. This conditional computation is what lets a model carry hundreds of billions of parameters while activating only a few billion per token, and it is the structural fact expert pruning exploits: if some experts are engaged rarely on a given distribution, removing them changes the model's behavior on that distribution little while reclaiming their share of memory.

**One-shot expert pruning.** A recent line of work removes whole experts in a single pass by scoring each expert's importance from its activation over a calibration set and dropping the least salient, without the iterative retraining that structured pruning classically requires. A parallel line (channel-pruning with a short recovery fine-tune) trades a lightweight retraining step for finer granularity. Both make architectural assumptions, about how the router exposes its decisions, how experts are laid out, and how activations can be observed, and those assumptions are exactly what a published method encodes for the architectures it was released against. Applying such a method to an unsupported architecture is therefore not a configuration change but an engineering port, which §3 documents.

**Post-training quantization and in-distribution calibration.** Low-bit post-training quantization compresses a trained model's weights with no gradient step, and importance-matrix calibration improves it by weighting the quantization error toward the activations a reference corpus actually produces. The series' prior result (Chapter 15) is that calibrating that matrix on the operator's own corpus, rather than a generic text sample, yields materially better low-bit fidelity on the operator's workload; this paper inherits that method and applies it after pruning.

**Multi-token prediction.** Some recent models ship an auxiliary head that predicts several future tokens at once, usable as a self-speculative drafter to raise decode throughput. When a model is pruned, that head must be handled deliberately (recovered, matched to the pruned trunk, and protected through quantization), which §7 addresses.

**Pre-registration and hash-sealed evaluation.** The evaluation discipline used here is the series standard: hypotheses, benchmark sets, decision bars, and analysis plans are sealed to a cryptographic hash before any measurement, so that a favorable result cannot be manufactured by post-hoc choice and an unfavorable one cannot be quietly dropped. §6 and §8 are both reported under it, which is why the FAIL in §8 is reportable at all.

### 3. Porting One-Shot Pruning onto an Unsupported Architecture

A published pruning method is a claim about a family of architectures, not a universal tool, and the gap between the two is where the engineering lives. The one-shot expert-saliency method this work builds on was released against a set of supported mixture-of-experts layouts; the target here, a 122B-parameter model with roughly 256 routed experts (eight routed plus one shared per token) over a gated-delta-network hybrid backbone, was not among them. This is not incidental: the method's own reference implementation ships hand-patched model definitions only for the specific architectures whose routers do not otherwise expose the signals it needs, so an unsupported target is a code change by construction rather than a configuration flag. Running the method as shipped was not an option, and the four adaptations it required are the substance of this section, because each is a place where a datacenter-shaped assumption met a single-operator, single-GPU reality and had to bend.

**An observer registry for an unsupported gate layout.** Expert-saliency pruning works by observing, over a calibration pass, which experts a workload actually routes to and how much each contributes; that observation has to attach to the model's specific gate and expert modules. The target's router exposed its routing through a gate whose shape and top-k selection did not match any registered hook, so the first task was to write an observer-hook configuration and register it against the architecture's own module names, teaching the method where the routing decision lives before it could measure anything.

**A routed-only saliency branch.** The reference path accumulates a dense per-expert activation tensor. For this model that tensor would have been shaped on the order of (experts × tokens × hidden), a materialization both infeasible on the available memory and, more importantly, unnecessary: the router already emits the routing weights directly. The branch was rewritten to read saliency from the router gate's own output rather than reconstruct it densely, which is both what made the pass fit and, arguably, the cleaner formulation for any sparsely-routed model.

**Offload by stash-swap of memory-mapped tensors.** The planned approach moved idle expert weights to a meta device between blocks. In practice the method that held resident memory flat was different: stash-swapping memory-mapped, CPU-backed tensors, which kept anonymous memory flat across the run in a way the meta-device path did not, at the cost of a reload the design accepts. This is recorded as a deliberate deviation, not an incidental one, because the anonymous-memory flatness it bought is what let the run survive on the available RAM.

**Streaming prune-and-save.** Assembling the full pruned model in memory before writing would have required on the order of 173 GB of anonymous memory, which the box does not have. The prune and the save were therefore interleaved, writing pruned blocks to disk as they were produced rather than holding the whole model, so that peak memory tracked a single block rather than the entire network. The output is a sharded, text-only checkpoint with a hash-sealed manifest.

None of these is exotic in isolation; together they are the difference between a method that runs in the environment it was written for and one that runs in a sovereign environment it was not. Porting a datacenter-shaped compression method onto an unsupported architecture, and onto hardware two orders of magnitude smaller than its assumed substrate, is itself a reportable contribution, and the deviations above are offered as reusable rather than incidental: any operator attempting the same class of port on a sparsely-routed model will meet the same four seams.

### 4. Pruning a 122B Model on One Card

Calibration came first. A general-reasoning calibration mix was assembled from the operator's registered datasets and validated to parse under the method's expected format, deliberately excluding coding, tool-use, and multiple-choice train-splits so that the pruning target was a research-reasoning workload rather than a benchmark-shaped one. The observer then ran over the model block by block, 48 in all, recording per-expert saliency from the router output as described in §3, and the least-salient experts were removed to take each layer from 256 experts to 192, a 25% reduction, with the shared expert preserved intact.

The run's defining constraint was that the 12 GB card never held the whole model. The block-wise path kept only the block under work resident, and expert weights not in use were stash-swapped to memory-mapped host tensors, so peak resident memory tracked a single block rather than the 122B network. The prune executed in roughly 23 hours and 17 minutes of wall-clock under a synthesis-lock (which defers heavy background work) and two-axis hardware telemetry that recorded zero thermal events across the run. One honest operational detail belongs on the record: an initial run drove anonymous host memory to a self-kill; the fix was a double-buffered input-replacement path that bounded the growth, after which the run completed cleanly. The pruned model was written by streaming save to a sharded, text-only checkpoint of roughly 186 GB with a hash-sealed manifest, and a verify pass confirmed it loaded by memory-map and produced finite output on a CPU forward. The single-card, block-wise, telemetry-locked prune of a 122B model is, to the operator's knowledge, not something the source method was demonstrated to do, and it is the practical core of the fit half of the paper's claim.

### 5. In-Distribution Calibration and Low-Bit Quantization

The pruned checkpoint was converted to a half-precision serving format and then quantized under the in-distribution discipline of Chapter 15, now applied one step downstream: the importance matrix was calibrated on the sovereign corpus so that the low-bit quantizer preserved precision where the operator's own workload places it. Computing that matrix directly on the half-precision model was measured at roughly 118 hours of wall-clock, impractical for an iterable pipeline; computing it instead on an 8-bit copy with a doubled micro-batch cut that to about 2 hours and 4 minutes, a 29-fold reduction, while covering all but one of the model's 9,216 expert slots. From the calibrated matrix the model was quantized to two tiers, a roughly 56.6 GB 4-bit and a roughly 66.3 GB 5-bit build, each verified to load and answer a health probe on the serving surface. The two-tier output is what let the quality gate of §6 test the pruned model at 4-bit against the unpruned model at the 2-bit setting its size forced, a comparison at matched memory rather than matched nominal precision.

### 6. The Served Quality Gate (pre-registered)

Pruning that preserves perplexity can still degrade the behavior an operator actually depends on, so the claim that matters is not a compression ratio but a served-quality comparison against the model the pruned one is meant to replace. That comparison was run as a pre-registered experiment: the hypotheses, the benchmark set, the decision bars, and the analysis plan were sealed to a hash before any measurement, ratified by the operator, and closed with a final seal (one bar was re-tiered by a logged amendment, itself sealed). The comparison is deliberately adversarial to the pruned model's favor: it pits the pruned model at 4-bit quantization against the unpruned model at the more aggressive 2-bit quantization it required to be served at all on the same card, so that both occupy comparable memory and the test asks whether pruning-then-lighter-quantization beats quantization-alone.

On a paired evaluation over the deployed serving surface, the pruned 4-bit model showed no measured loss against the unpruned 2-bit baseline on any axis. Pooled mathematics came in 2.5 percentage points higher (67.5 versus 65.0, n = 80), the graduate-level knowledge benchmark 1.01 points higher (41.9 versus 40.9), and MATH-500 tied at 94.0. The direction of every delta is favourable, but the honest reading is parity rather than improvement: McNemar’s test returns p = 0.75, 0.91 and 1.00 respectively, so not one of these differences is statistically distinguishable from zero, and the largest of them, AIME24 at +6.67 points, rests on four discordant pairs against two. The pre-registered claim was retention, and retention is what the data support. The finding a reader should take away is that removing a quarter of the expert weights cost nothing measurable, not that it helped. Against an 8-bit reference, the pruned 4-bit model cleared all five pre-registered fidelity axes (mean, tail, and 95th-percentile Kullback-Leibler divergence, perplexity delta, and top-1 agreement), with a mean KL divergence of 0.0240 and 93.2% top-1 agreement, indicating it tracked the high-precision reference closely. The in-distribution character of the prune shows in the perplexity split: the pruned model's perplexity on a held-out sovereign corpus sat only about 2.7% above the unpruned model's, against roughly 35% on generic web text, which is exactly the asymmetry the thesis predicts once the experts removed are the ones the sovereign workload never exercised. Throughput did not pay for the fidelity: the pruned 4-bit model served at 16.65 tokens per second against the unpruned 2-bit model's 15.60. The headline is therefore two quantization tiers of better fidelity at equal-or-better speed, and on that result the pruned model replaced the unpruned one as the operator's served research model. Three facts about how the gate was run belong in the record. It was not a small run: 556 unique cells across 598 records plus a seven-run fidelity battery, roughly 46 hours of wall clock across four fires, with both arms pinned to the same server build, the same deployment preset, the same sampling parameters and seed, the same benchmark banks by manifest hash, and the same grader, and with each arm’s weights hash-verified before every fire. It was truncation-bound: generations hit the token budget on essentially every graduate-knowledge item and on roughly nine in ten competition-mathematics items, in both arms alike, so the absolute scores understate both models symmetrically and only the paired delta carries signal. And it was not clean. One fidelity bar was re-tiered before any primary data existed, by an amendment the operator ratified and the experiment sealed alongside the original registration; then, mid-run, 45 cells of the unpruned arm were poisoned when the collaborator killed a server under a mis-diagnosed driver fault. Those cells were quarantined rather than deleted, the byte-exact backup and the now-invalid summary were both kept on disk and labelled as such, the 45 cells were re-run clean, and the primary hypothesis was adjudicated only on the uncontaminated set. The failure is in the record because a pre-registration that survives only a clean run is not worth much.

| Benchmark bank   | n (paired) | Pruned Q4\_K\_M | Full UD-Q2\_K\_XL | Δ (pp) | McNemar p     |
| ---------------- | ---------- | --------------- | ----------------- | ------ | ------------- |
| AIME24           | 30         | 23.33%          | 16.67%            | +6.67  | 0.6875        |
| MATH-500         | 50         | 94.00%          | 94.00%            | 0.00   | 1.000         |
| GPQA (diamond)   | 198        | 41.92%          | 40.91%            | +1.01  | 0.9088        |
| Pooled math (H1) | 80         | 67.50%          | 65.00%            | +2.50  | 0.7539 · PASS |

***Table 1.** Paired served retention, pruned 4-bit versus unpruned 2-bit, on the deployed surface (pre-registered served quality-gate experiment dated 2026-07-04, final seal 2026-07-06, thinking-enabled). Accuracies are exact-match; McNemar's test is on the discordant pairs. Pooled math (AIME24 + MATH-500, n = 80) is the pre-registered primary hypothesis H1. Discordant pairs ran 4 against 2 on AIME24, 2 against 2 on MATH-500, and 39 against 37 on GPQA, and the Wilson 95% intervals overlap heavily on every bank (GPQA pruned \[35.3%, 48.9%] against full \[34.3%, 47.9%]), which is why the verdict is stated as retention rather than gain.*

| Quant tier         | KL mean | KL p95 | KL max | Top-1 agree | PPL Δ% |
| ------------------ | ------- | ------ | ------ | ----------- | ------ |
| Q4\_K\_M (served)  | 0.0240  | 0.0882 | 3.227  | 93.21%      | 1.47%  |
| Q5\_K\_M           | 0.0124  | 0.0457 | 3.668  | 95.09%      | 0.31%  |
| Pre-registered bar | ≤ 0.05  | ≤ 0.5  | ≤ 7.6  | ≥ 90%       | ≤ 5%   |

***Table 2.** Distributional fidelity of the quantized pruned model against a pruned 8-bit reference (100 chunks; llama.cpp KL-divergence tooling; args -ngl 99 --n-cpu-moe 48 -b 2048 -ub 2048 -c 512). Both tiers clear all five pre-registered fidelity axes (H3 fidelity gate); the served Q4\_K\_M tier passes with margin.*

| Arm                 | WikiText-2 PPL (generic) | Held-out sovereign PPL |
| ------------------- | ------------------------ | ---------------------- |
| Pruned Q4\_K\_M     | 6.9618                   | 12.4977                |
| Full UD-Q2\_K\_XL   | 5.1404                   | 12.1696                |
| Pruned / full ratio | 1.354 (+35.4%)           | 1.027 (+2.70%)         |

***Table 3.** Perplexity of the pruned model against the unpruned baseline on generic web text versus a held-out draw of the operator's sovereign corpus. The pruned model pays a large perplexity penalty on the generic distribution and almost none on the sovereign one, the signature of an in-distribution prune.*

![Figure 1](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2F2LQ7vJaOJiQNmaF8Dn5H%2Fgb-ch17-fig1-ppl-asymmetry.png?alt=media)

***Figure 1.** The in-distribution result, visualized. Removing the experts the sovereign workload never routes to costs a third of the model's perplexity headroom on generic text but essentially nothing on the workload it was pruned for. This asymmetry is the thesis of the paper, measured on held-out data.*

### 7. Recovering the Multi-Token-Prediction Head and Vision

#### 7.1 Recovering the multi-token-prediction head

The base model ships a multi-token-prediction block that is itself a mixture-of-experts, usable as a self-speculative drafter to raise decode throughput, and pruning the trunk left it behind. It was recovered from the base weights through the same observer extension used for the trunk, pruned from 256 experts to 192 so its expert count matched the pruned model, and grafted back on, after which the combined model was re-converted and re-quantized. One detail carried from a prior cycle governed the quantization: the multi-token-prediction tensors were protected from the most aggressive bit-width, because an earlier finding showed 4-bit on those tensors collapses the draft-accept rate that makes the head worth having. The grafted model was smoke-tested on the serving surface for draft acceptance and throughput against the non-multi-token baseline.

#### 7.2 Enabling vision

The base 122B is a full vision-language model, and a useful property of expert pruning is that it left that capability reachable. Vision enters the network at the hidden-embedding interface, downstream of the multimodal projector and upstream of the expert bank, so pruning experts did not touch the path a vision input takes into the model; the pruned text trunk needed no rework to accept images. The multimodal projector was extracted from the base and paired with the pruned trunk, and the combination was smoke-tested on real inspection frames from the operator's workflow. One risk is recorded honestly: the pruning saliency was calibrated on a text-only corpus, so experts that matter mainly for vision-conditioned routing could have been among those removed; whether vision-conditioned quality holds under the text-calibrated prune is an open question the deployment continues to probe rather than one this paper closes.

### 8. A Replication That Failed, Reported in Full

The 122B result is not the whole story, and reporting only the experiment that worked would misrepresent the method. A parallel pre-registered attempt to prune a 35B-class model to a fully-resident, on-box-trainable size returned a composite FAIL, and it is reported here in full because the failure is more instructive than the success. The pre-registration crossed method (a channel-pruning-plus-recovery approach as primary, the expert-saliency method as comparator) against ratio, multi-token-prediction handling, and quality; pruned checkpoints at roughly 25% and 31% reduction were produced and recovery-fine-tuned on-box, with the recovery run itself a useful datum (a 24.6B-parameter model trained at about 10.3 GiB resident on the 12 GB card).

The composite verdict failed on the serving-speed hypothesis, and it failed on a real hardware signature rather than a modeling artifact. The recovered model at the chosen quantization measured larger than the card's dedicated memory; the driver spilled the overflow into shared system memory, and that spill dragged sustained throughput to roughly 8.5 tokens per second against a bar of 63.8. The spill signature is exactly what a fully-resident-serving hypothesis is meant to catch, and it caught it.

The deeper finding was a reframe, not a data loss. On review, the experiment had calibrated a model destined for the Architect role using the research model's calibration mix, which is the wrong axis for a role whose governing doctrine is a single sovereign corpus cited at every layer of the compression chain. The frozen verdicts stand as recorded (the incumbent model was retained), but their correct reading is that the experiment measured a corpus-slot mismatch as much as a compression limit, and it was the pre-registration discipline that made that separable rather than a silent confound. A matched-corpus re-run was pre-registered in response, carrying a corpus-identity clause that binds one hash-sealed sovereign draw across every layer of the chain (calibration, recovery, quantization, and any subsequent training), with the mismatched cells retained as the explicit control. A method paper that showed only the passing experiment would be less trustworthy than one that shows both; the failing experiment is what calibrates the reader's confidence in the rest.

| Gate                | Pre-registered rule                               | Measured (at seal)                                 | Verdict |
| ------------------- | ------------------------------------------------- | -------------------------------------------------- | ------- |
| H1 math             | Δ pooled-math ≥ −5.0 pp (n=80)                    | 66.25% vs 67.50% = −1.25 pp (p=1.0)                | PASS    |
| H2 knowledge        | Δ GPQA-198 ≥ −6.0 pp                              | 45.45% vs 29.80% = +15.66 pp (truncation-salvaged) | PASS    |
| H3 resident serving | resident (no offload), ≤ 12,227 MiB, ≥ 63.8 tok/s | 12.68 / 13.69 GiB > card; spill; 8.82 / 8.48 tok/s | FAIL    |
| Composite           | H1 ∧ H2 ∧ H3                                      | H3 forecloses                                      | FAIL    |

***Table 4.** The composite verdict of the first 35B experiment on its frozen pre-registered rules (experiment dated 2026-07-06, final seal 2026-07-12). In that experiment quality was retained (H1, H2); the model simply would not fit resident at a servable quantization on the 12 GB card (H3), and under the conjunctive rule H3 forecloses the composite. The incumbent model was retained. The matched-corpus re-run reported later in this section (sealed 2026-07-14) reached a different verdict on different grounds, failing on mathematics retention rather than on serving.*

The H3 failure is worth showing rather than only stating, because its shape is a lesson about consumer hardware that a datacenter result would never surface. The recovered model's throughput did not degrade smoothly as it approached the memory ceiling; it inverted. As expert layers were moved back onto the card (fewer offloaded), throughput improved as expected, until the model crossed the point where the driver silently paged the overflow into shared system memory, at which point sustained throughput collapsed by 85% even though the reported dedicated-VRAM figure still sat under the nominal budget. That last detail, dedicated VRAM reading 11,829 MiB against a 12,227 MiB budget while real throughput cratered, is the trap: the obvious instrument lies, and only the throughput measurement catches the spill.

![Figure 2](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2F6waXYnH9bhj3NGpnZz5p%2Fgb-ch17-fig2-placement-inversion.png?alt=media)

***Figure 2.** The placement inversion, measured on the 35B self-speculative draft path. Moving experts onto the card raises throughput (left, center) until the model crosses the spill threshold, where it collapses (right) despite a dedicated-VRAM reading still under budget. This is the "fully-resident serving" hypothesis earning its place: on consumer hardware the fit constraint is a cliff, not a slope.*

One receipt from the failed experiment stands independent of the corpus mismatch and de-risks the training work of §9: the recovery fine-tune itself ran on-box, training the 24.6B-parameter pruned model at roughly 10.3 GiB resident on the 12 GB card at about 250 seconds per step, lowering validation cross-entropy by 0.26 nats. The model that would not *serve* resident nonetheless *trained* resident, which is the exact envelope §9 sets out to exploit.

The matched-corpus re-run promised above has since completed, sealed 2026-07-14, and its result is the more important of the two. That chain ran end to end on frozen forks, under a pre-registration whose canonical hash re-verified exactly at seal: one hash-identical sovereign draw carried through calibration, a 26.6B-parameter cut, a 200-step on-box recovery that moved validation cross-entropy from 3.418 to 2.040, matched-corpus quantization, and a paired evaluation of 556 items with zero errors over 14.7 hours. It also failed, and it failed on the axis the first experiment had passed. Graduate-level knowledge rose sharply, 18.18 points above the deployed baseline (47.98% against 29.80%, p below 0.0001), while pooled mathematics fell 11.25 points below it (56.25% against 67.50%, p = 0.0225), driven almost entirely by MATH-500 falling from 98% to 84%. Under the conjunctive rule the mathematics gate forecloses the composite, and the incumbent model was retained.

The direction of those two movements taken together is the finding. Knowledge rose against both the deployed baseline and the pre-registered mismatch control, and mathematics fell against both. That is not the signature of a corpus that underperforms; it is the signature of a corpus placed at the wrong layer. The experiment was therefore concluded at the honest failure without running its remaining stages, on the reasoning that its premise had been invalidated by a reframe rather than by data, and the seal names three errors plainly. The axis was wrong, because specialization is properly measured on tool-calls-to-objective and typed behavioral restraint rather than on capability benchmarks. The layer was wrong, because a role-flavored corpus was pushed into the base slot, against an architecture that splits base, adapter, and behavioral reflex and assigns role specialization to the adapter. And the model was wrong for that corpus's layer. A broken premise cannot be repaired by amendment, since an amendment could swap the model and still leave mathematics retention standing as the gate; concluding cleanly at the failure is the pre-registration discipline working rather than failing. One guardrail is sealed into the record against the opposite reading, and it belongs here too: the incumbent was kept because it is the correct broad reasoning substrate for that role, the mismatched cells were the pre-registered control and never a deployment candidate, and any reading of this result as the sovereign corpus underperforming is false. What the mathematics gate caught, correctly, was a base going too narrow.

The reframe has a constructive tail, because it turns the failure into a method. The matched-corpus recovery ran at zero replay: entirely sovereign-domain data, with no general-corpus tokens mixed back into the continued-pretraining batches. The continued-pretraining literature is close to unanimous that this is the one setting that craters general capability, since catastrophic forgetting is driven by unbalancing the corpus rather than by the presence of domain data, and the standard mitigation is to replay on the order of 15 to 30% general tokens into every batch, with even 1 to 5% measurably helping, at a learning rate several-fold below the original pretraining rate. Read against that literature, the degradation is a predicted endpoint rather than an anomaly. It has two independent sightings across the two 35B experiments, and they were taken on different instruments.

The first sighting is in the matched-corpus experiment, and it reaches the sealed verdict itself: pooled mathematics fell 11.25 points and failed its bar, while graduate-level knowledge rose 18.18 points. That is the zero-replay endpoint of the mixture question, measured on the pre-registered instrument.

The second sighting is in the first experiment, the 35B prune-and-drafter run whose composite verdict Table 4 reports, and it does not reach that experiment's sealed thinking-enabled verdict, which retained both mathematics and knowledge. It is plain instead in an exploratory no-think deployment read, where the two mathematics banks part company. AIME24 falls 53.3 points, the largest single fall reported anywhere in this paper, while MATH-500 holds up best of everything measured, and graduate-level knowledge falls 24.7 points between them, both falls significant. That divergence is the textbook signature of catastrophic forgetting driven by corpus imbalance rather than by the presence of domain data, and although an exploratory read moves no sealed verdict, it converts the corpus-slot mismatch named above from a diagnosis into a quantity: the fix is a replay ratio, not a hope.

The ratio need not be found by blind grid search. The domain-specific continued-pretraining scaling law of Que et al. (arXiv:2406.01375), validated on Qwen-family models across code and mathematics domains, predicts validation loss at any mixture ratio from a handful of cheap small-model runs, and collapses to the Chinchilla law when the ratio is held fixed. Its critical-mixture-ratio companion (arXiv:2407.17467) reads the elbow between general retention and domain transfer directly off the fitted curve, and a cross-domain extension derives the law for a new domain such as the sovereign corpus at roughly one percent of a full sweep's cost. The matched-corpus experiment sits at the zero-replay end of that curve, and its mathematics loss is the operator's own first measured point on it.

The first experiment differed from the matched-corpus experiment in the way the curve would predict, because its recovery corpus was not sovereign-only. The amendment record sealed with that first experiment lists general reasoning material among its recovery data, including a mixture-of-thoughts mathematics set and a personas-mathematics set. Those same cells were afterwards re-classed as the pre-registered mismatch control for the matched-corpus re-run, which is why they appear twice in this section wearing two roles: the corpus composition that made them a useful control is the same composition that makes the following comparison possible.

Compared on the sealed verdicts, which are the like-for-like measurement, pooled mathematics was retained in the first experiment at 1.25 points down and failed its bar in the matched-corpus experiment at 11.25 points down, both against the same 5.0-point bar and the same baseline. Reasoning replay is one difference between the two. They also differ in the corpus-identity chain that the re-run was built to change, binding a single hash-sealed sovereign draw across calibration, recovery and quantization where the first experiment did not, so this comparison indicates a direction rather than isolating a cause. The exploratory no-think figures quoted earlier were taken on a different instrument and are not what the comparison rests on.

The bank-level divergence is a second reason the replay composition cannot be claimed as a clean explanation. The replayed material was mathematics-flavored rather than general, which is consistent with MATH-500 holding, but it plainly did not protect AIME24, whose competition-style problems behave here like a general-reasoning axis rather than like the bank they are pooled with in §6. MATH-500's resilience is equally consistent with its being the calibration-dense domain, and on two experiments and one exploratory read this paper cannot separate calibration density from replay composition. Both readings are recorded and neither is asserted.

Two experiments are an observation rather than a fitted curve, and one caveat from the first experiment's amendment record travels with the comparison. The reasoning material in that recovery corpus may share problem-lineage provenance with the mathematics banks used to score both experiments; and within each paired measurement only the pruned cell saw recovery data, while the deployed baseline it was scored against did not. The first experiment's retention figure therefore carries an exposure the matched-corpus figure does not. The comparison motivates the replay program rather than standing in for it.

### 9. The On-Box Training North Star (active)

This section is active work, and is written as such: a pre-registered design with interim receipts, not a result. It is also the reason the rest of the arc exists. The goal is to fine-tune the pruned big model on-box, on the operator's own hardware, through aggressive expert-offload low-rank adaptation: quantize to 4-bit, freeze the expert bank and offload it to the 128 GB of system memory, and keep resident on the card only the attention projections and the low-rank adapters that actually receive gradients, so that offload never touches the gradient path and resident memory stays inside the 12 GB envelope. It is the training-side mirror of the CPU expert-offload that already makes these models *servable* on the card, and pruning is the enabler that shrinks the model far enough for the training-time memory arithmetic to close.

The open question is a toolchain one: whether the purpose-built CPU-GPU expert-scheduling stack that the offload requires builds and runs on the operator's Blackwell-class GPU, against a faster alternative stack that omits offload and therefore falls short on memory. That gate is the first deliverable and is not yet passed. What de-risks the design is a receipt already in hand from the 35B recovery of §8: a 24.6B-parameter model was fine-tuned on-box at roughly 10.3 GiB resident on the 12 GB card, which demonstrates the resident-memory envelope is real rather than hoped-for. Whether the same approach reaches the pruned 122B model (whose active-parameter count after pruning is materially tighter, though still larger than the 35B case) is documented as reachable-or-not with explicit memory arithmetic when the toolchain gate resolves, and is not asserted here. The honest status is that the fit and quality claims of §4 through §8 are sealed, and the trainability claim that motivates them is engineering-gated and open.

### 10. Limitations

The limitations are stated plainly, in the register the series keeps. This is single-operator, single-hardware work: the results hold for one workload on one class of card, and the pruning is calibrated to that workload by design. That design is itself the central tradeoff: in-distribution pruning buys sovereign-workload fidelity and memory by giving up breadth on workloads the operator does not exercise, and a model pruned to one operator's distribution should not be expected to generalize to another's. The 35B FAIL of §8 is the guardrail on that tradeoff, the case where the pruning target and the role's true corpus were mismatched and the pre-registration caught it, and it should be read as a bound on the method, not an aside. The aggressive-ratio regime was left unexplored on purpose: a 50% reduction was pre-registered as code-only-lossless but unproven for general reasoning, and that cliff was not crossed here. Vision-conditioned quality under a text-calibrated prune is an open risk (§7.2). And the trainability claim that motivates the whole arc is unproven pending the toolchain gate of §9. None of these is fatal to the fit-and-quality result, and all of them are the reason this is a preliminary report rather than a closed one.

### 11. Conclusion

A 122B mixture-of-experts shipped for datacenter service now runs for a single operator, on a single 12 GB card, pruned to the experts that operator's workload actually uses and quantized under a matrix calibrated on that operator's own corpus, at higher fidelity than the quantization-only alternative it replaced and at equal-or-better speed. Getting there required porting a published pruning method onto an architecture it did not support, pruning a model far larger than the card that pruned it, and subjecting the result to a pre-registered gate it had to pass on the deployed surface rather than in a notebook. The one replication that failed is reported beside the one that worked, because a compression method is only as trustworthy as its worst honestly-measured case. What remains open, on-box training of the pruned model, is the claim the arc was built to reach, and it is left open rather than asserted. The through-line to the series is the same one that runs through the imatrix and domain-pruning papers: sovereignty is not only a matter of where the weights live but of whose distribution shapes them, and the operator's own corpus is the instrument that decides, at the architecture layer as at the quantization layer, what to keep.

### System Update: July 2026 (appended; the sealed body above is unmodified)

**The replacement happened, and has held.** The gate of §6 cleared the pruned model to replace the served research brain; the serving preset was repointed from the full 2-bit build to the pruned 4-bit build on 2026-07-06, the day that gate sealed, and it has served in that role continuously since. The superseded entry was left in place commented out rather than deleted, so the substitution is legible in the configuration itself. Nineteen days of residency at the time of writing is an availability record and not a quality claim; what it establishes is that the deployment decision in §6 was executed rather than merely recommended.

**Throughput has improved since the seal, by tuning rather than by any change to the model.** The preset records an operational note that a lower expert-offload setting sustains roughly 18.5 tokens per second within the same memory envelope, above the 16.65 measured at the gate, with a spill cliff one step further down that drops throughput to roughly a fifth of it. That figure is a deployment configuration note, not a sealed measurement, and it is reported here as such: it did not pass through the pre-registered instrument, and the sealed comparison of §6 stands unchanged. Its interest is that the offload boundary the arc was built around turns out to have had headroom left in it.

**The vision risk of §7.2 is now partly exercised, and remains open.** A second preset serves the same pruned weights with a multimodal projector, in a finished-video inspection role. That is production exercise of a text-calibrated prune on vision-conditioned work, which is the risk §7.2 names, and it has not produced a spill or a rollback. It is not a measurement. This paper reports no vision quality result for the pruned model, and nothing in this deployment closes §7.2, which stays open exactly as written. Deployment without incident is weaker evidence than a passed gate, and is offered only as the weaker thing.

***

*The Sovereign Stack · Sovereign Big-Model Compression · Chapter 17 · Part IV · v1.0.1 · License CC BY 4.0 · © Jamey Kistner, OSINTelligence LLC*

**Citation (preferred):** Kistner, J. (2026). *Sovereign Big-Model Compression: Porting One-Shot Expert Pruning onto a 122B Mixture-of-Experts and Serving It, Trainable, on a Single Consumer GPU*, version 1.0.1. OSINTelligence LLC.

*The reference list and provenance follow as a sub-page of this chapter.*