> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/part-iv-the-evidence-what-worked/17-sovereign-big-model-compression/the-quick-version.md). # The quick version **Continue the tour →** [Next: 18 · Corpus-Sovereign Self-Distillation, the quick version](/osintelligence/part-iv-the-evidence-what-worked/18-corpus-sovereign-self-distillation/the-quick-version.md) The short version of Chapter 17, three ways: the video walks the argument in a few minutes, the deep dive talks it through at a listening pace, and the infographic holds the whole chapter in one view. The full result, with the 122B mixture-of-experts pruned onto one 12 GB card at parity and the parallel 35B attempt reported as a composite failure, lives in the chapter itself: [17 · Sovereign Big-Model Compression](/osintelligence/part-iv-the-evidence-what-worked/17-sovereign-big-model-compression.md). {% embed url="" %} **The deep dive.** A podcast-style audio conversation about this chapter: two AI hosts walk through the argument, the incidents behind it, and what it means, at a listening pace. Generated in Google's Gemini LM (formerly NotebookLM) from the chapter itself; the link opens the audio on Google's site. {% embed url="\_" %} *The conversation is AI-generated: an interpretation of the chapter, not the chapter. It can compress, paraphrase, or get details wrong. The written chapter is the authoritative, canonical source:* [*17 · Sovereign Big-Model Compression*](/osintelligence/part-iv-the-evidence-what-worked/17-sovereign-big-model-compression.md)*.* *** ![The Sovereign Big-Model Compression, the chapter in one view.](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2F5MIE1Rg8lHETKha9qCRz%2Fbmc-infographic.png?alt=media) *** ### Chapter notes Section-by-section notes in two registers: the technical note on the left, the same idea in plain language on the right. Every row is one idea, so you can read straight across from one register to the other. The technical terms stay visible in the plain column on purpose; they are the vocabulary worth keeping. #### Abstract **The point:** the flagship arc: a datacenter-class 122B model, pruned to the operator's workload, served on one 12 GB card at better fidelity than the alternative it replaced. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The frontier is very large mixture-of-experts models whose total parameter count sits far above consumer memory even though few experts activate per token; quantization alone cannot close the gap for a 122B on a 12 GB card, and does nothing for on-box trainability; the paper takes the architecture-side route: one-shot expert pruning calibrated on the operator's workload. | The best open models are now too big for any consumer card even in compressed form (**the fit gap**), and compression alone also leaves no room to train. The answer here is surgery instead (**expert pruning**): remove the specialists this operator's work never calls on. | | The full arc is reported: porting a published saliency method onto an unsupported architecture (four documented adaptations), pruning block-wise so the card never holds the whole network, calibrating the importance matrix on the sovereign corpus, quantizing to two tiers, and passing a pre-registered served-quality gate against the unpruned model. | The chapter walks the whole road (**the full arc**): adapt a published method that did not support this model, prune a 122B network on a card that cannot even hold it, tune the compression on home data, and then prove the result on the live serving surface under sealed rules. | | The pruned 4-bit model matches or beats the unpruned 2-bit on pooled math, graduate knowledge, and a held math set at equal-or-better throughput, so it replaced the unpruned build in production; a parallel 35B replication returned a composite FAIL, reported in full; the on-box training goal is active work, reported as design with interim receipts, not a result. | The verdict: the pruned model won its head-to-head and took over production (**the replacement**). The parallel experiment that failed is printed beside it in full, and the ultimate goal, training the big model on the desk PC, is honestly labeled unfinished. | #### 1. Introduction **The point:** two gaps, fit and trainability, and the sovereign observation that makes them tractable: one operator does not need every expert. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MoE sparsity makes big models fast to run but not small to hold: the full expert bank must be resident or paged; a 122B-class model must be served at an aggressive 2-bit setting that costs quality, and leaves no headroom for gradients. | The trick that makes giant models fast (**activation sparsity**: only a few specialists wake per word) does not make them small: every specialist still has to live somewhere. Forcing the giant onto the card meant the harshest compression tier, and training was out of the question entirely. | | Any one operator's workload exercises a stable subset of experts; in-distribution pruning removes the rarely-routed rest, calibrated by the operator's own corpus: the architecture-side mirror of the Chapter 15 calibration thesis, and the two compose. | The unlock (**the sovereign angle**): this operator's work routes to a stable subset of the specialists. Delete the ones it never calls, chosen by watching real home traffic, and compress what remains. Chapter 15 tuned the compression with home data; this chapter chooses the architecture with it. | | Seven contributions, each tied to a sealed receipt or a flagged-active phase: the port, the block-wise single-GPU prune, the calibration-and-quantization pipeline, the served gate, the fully-reported 35B FAIL, the head-and-vision recovery, and the active training design. | Seven pieces, each labeled sealed or active (**receipts or flags**): nothing in the list floats without one or the other. | #### 2–3. Background and the Port **The point:** a published method is a claim about the architectures it shipped for; making it run here took four documented adaptations, offered as reusable seams. | The technical note | In plain language | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | One-shot expert pruning scores each expert's importance from activations over a calibration set and drops the least salient without iterative retraining; the method's reference implementation ships hand-patched definitions only for specific architectures, so an unsupported target is a code change by construction. | The base method is published science (**one-shot expert pruning**): watch which specialists matter on sample traffic, cut the rest, no retraining loop. But its released code only understands certain model families; this one was not among them (**an engineering port, not a configuration flag**). | | The four seams: an observer registry for an unsupported gate layout (teaching the method where the routing decision lives), a routed-only saliency branch (reading importance from the router's own output instead of materializing an infeasible experts × tokens × hidden tensor), offload by stash-swap of memory-mapped tensors (kept anonymous memory flat where the planned meta-device path did not), and streaming prune-and-save (writing pruned blocks to disk instead of holding \~173 GB in memory). | The four adaptations (**the porting seams**): teach the tool where this model makes its routing decisions, read importance from what the router already reports instead of rebuilding a tensor too big to exist, park idle weights in a memory-mapped stash that provably kept RAM flat, and write results block-by-block because the assembled model would need 173 GB the box does not have. | | Each seam is a place where a datacenter-shaped assumption met a single-operator single-GPU reality; the deviations are offered as reusable method: any operator porting this class of method onto a sparsely-routed model will meet the same four. | The port itself is the contribution (**reusable seams**): these are not workarounds but the general checklist for anyone bringing datacenter compression tools down to a desk. | #### 4–5. The Prune and the Quantization **The point:** 48 blocks, 256 experts cut to 192 per layer, the card never holding the whole model, and a 29× calibration shortcut that made the pipeline iterable. | The technical note | In plain language | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The calibration mix deliberately excluded coding, tool-use, and multiple-choice train-splits so the pruning target was a research-reasoning workload, not a benchmark-shaped one; the observer ran block-by-block over 48 blocks, cutting each layer from 256 to 192 experts (25%), shared expert preserved. | The traffic sample was chosen to represent real work, not test-taking (**not benchmark-shaped**), and a quarter of the specialists were cut layer by layer, sparing the one specialist every token visits (**the shared expert**). | | The defining constraint: the 12 GB card never held the whole model; \~23 h 17 m wall-clock under a synthesis-lock and two-axis telemetry, zero thermal events; one honest operational detail on the record: an initial run drove host memory to a self-kill, fixed by a double-buffered input-replacement path; the output is a \~186 GB sharded text-only checkpoint with a hash-sealed manifest. | The surgery ran a full day on a card fifteen times smaller than the patient (**block-wise pruning**), cleanly, with one recorded stumble: the first attempt ate all the system memory and killed itself, and the fix is documented rather than hidden. | | Quantization followed Chapter 15's in-distribution discipline one step downstream; computing the importance matrix on the half-precision model measured \~118 h, cut to \~2 h 04 m (29×) by computing on an 8-bit copy with doubled micro-batch, covering 9,215 of 9,216 expert slots; two tiers produced: \~56.6 GB 4-bit and \~66.3 GB 5-bit. | The compression tuning had a practicality problem, five days of compute, solved by running the measurement on an already-compressed copy (**the 29× shortcut**) with near-total coverage. Two sizes were built so the fair fight of the next section could happen at matched memory. | #### 6. The Served Quality Gate **The point:** the deepest water: the pre-registered head-to-head on the live surface, the retention-not-gain honesty, and the perplexity split that is the whole thesis. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The comparison is deliberately adversarial to the pruned model's favor: pruned 4-bit vs unpruned 2-bit at comparable memory, asking whether pruning-then-lighter-quantization beats quantization-alone; hypotheses, benchmarks, bars, and analysis sealed before measurement, one bar re-tiered by a logged, sealed amendment. | The fight was arranged at matched weight (**matched memory, not matched precision**): the pruned model kept gentler compression, the full model needed the harshest tier just to fit. The rules were sealed before any scores existed. | | Results: pooled math +2.5 pp (67.5 vs 65.0, n=80), GPQA +1.01 (41.9 vs 40.9), MATH-500 tied at 94.0; every delta favourable but the honest reading is parity: McNemar p = 0.75/0.91/1.00, no difference distinguishable from zero; the pre-registered claim was retention, and retention is what the data support. | Every number leaned the pruned model's way, and the chapter refuses the flattering headline anyway (**retention, not gain**): the statistics say the differences are indistinguishable from zero, so the claim is "removing a quarter of the experts cost nothing measurable," not "it helped." | | Against an 8-bit reference the served 4-bit tier cleared all five pre-registered fidelity axes (mean KLD 0.0240, top-1 93.2%); the perplexity split carries the thesis: +2.7% on the held-out sovereign corpus vs +35.4% on generic web text; throughput 16.65 vs 15.60 tok/s; on that result the pruned model replaced the unpruned one in production. | The signature measurement (**the perplexity split**): the pruned model pays a heavy penalty on generic internet text and essentially none on the work it was pruned for. That asymmetry is the whole idea, measured on held-out data, and it is why the pruned model took the production seat. | #### 7. Recovering the Head and Vision **The point:** the two capabilities pruning left behind, one recovered and matched, one enabled with its risk stated open. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The multi-token-prediction block (itself a MoE, usable as a self-speculative drafter) was recovered from base weights via the same observer extension, pruned 256 → 192 to match the trunk, grafted back, and re-quantized with its tensors protected from the most aggressive bit-width, per a prior finding that 4-bit collapses draft acceptance. | The model's built-in speed booster (**the drafting head**) was recovered, trimmed to match the pruned trunk, and reattached, with one hard-won rule honored: compress that part too hard and the booster stops boosting. | | Vision enters at the hidden-embedding interface, downstream of the projector and upstream of the expert bank, so the pruned trunk needed no rework to accept images; the projector was extracted and smoke-tested on real inspection frames; the open risk is stated: saliency was text-calibrated, so vision-relevant experts could have been removed, a question the deployment probes rather than closes. | The picture-understanding half survived untouched, because images enter upstream of everything pruned (**vision by architecture**). One risk is left honestly open: the pruning watched only text traffic, so specialists that matter for images could theoretically have been cut. | #### 8. A Replication That Failed, Reported in Full **The point:** the parallel 35B attempt: a real hardware cliff where the instrument lies, a corpus placed at the wrong layer, and a failure that became a method. | The technical note | In plain language | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The 35B prune-to-resident attempt returned a composite FAIL on H3: the recovered model measured larger than dedicated memory, the driver spilled into shared system memory, and throughput collapsed to \~8.5 tok/s against a 63.8 bar; the trap is the instrument: dedicated VRAM read 11,829 MiB against a 12,227 MiB budget while real throughput cratered; the placement inversion is shown in Figure 2: a cliff, not a slope. | The failed experiment failed on real physics (**the spill cliff**): past a hidden threshold the driver quietly pages memory and speed collapses by 85%, while the obvious memory gauge still reads under budget (**the instrument lies**). Only the throughput measurement catches it. | | The deeper finding was a reframe: the experiment calibrated an Architect-role model on the research model's mix, a corpus-slot mismatch the pre-registration made separable; a matched-corpus re-run (sealed 2026-07-14) with one hash-identical draw across the whole chain also failed, on the opposite axis: GPQA +18.18 pp, pooled math −11.25 pp (MATH-500 98% → 84%), the signature of a corpus placed at the wrong layer, not a corpus that underperforms; the seal names three errors plainly (wrong axis, wrong layer, wrong model for that layer) and the guardrail is sealed against misreading: the sovereign corpus did not underperform, the base went too narrow. | Two failures, each teaching (**the reframe**): the first run mixed up which data belongs to which role, and the sealed re-run with the right paperwork failed differently, knowledge way up, math down, which is the textbook shape of a specialist corpus loaded into the generalist slot (**the mis-layered corpus**). The record explicitly forbids the lazy reading: home data did not fail, the placement did. | | The constructive tail: the matched-corpus recovery ran at zero replay (no general tokens mixed in), the one setting the continued-pretraining literature is near-unanimous craters general capability; the published scaling law (Que et al.) predicts the fix, a replay ratio read off a fitted curve at \~1% of a sweep's cost; the zero-replay math loss is the operator's first measured point on that curve; one receipt stands independent: a 24.6B model trained on-box at \~10.3 GiB, the exact envelope §9 exploits. | The failure converts into a recipe (**zero replay, named**): retraining on purely specialist data is the known way to erode general skill, the published cure is mixing a measured fraction of general text back in, and this run is now the first data point on that curve. And buried in the wreck is the arc's best receipt: a 24.6-billion-parameter model genuinely training on the 12 GB card. | #### 9–10. The Training North Star, and Limitations **The point:** the goal the arc was built for is honestly open, and the limits are stated as bounds, not asides. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The design: quantize to 4-bit, freeze and offload the expert bank to 128 GB system memory, keep resident only the attention projections and the low-rank adapters receiving gradients, so offload never touches the gradient path; the open gate is toolchain (whether the CPU-GPU expert-scheduling stack builds on Blackwell); de-risked by the 24.6B on-box receipt; explicitly not asserted for the 122B until the gate resolves. | The end goal (**on-box training of the giant**) has a written design and one strong proof-of-envelope, and the chapter still refuses to claim it: a specific software gate has to pass first, and until it does the claim stays open. | | Limitations in the series register: single operator, single hardware class; in-distribution pruning trades breadth for sovereign fidelity by design (a model pruned to one operator's distribution should not generalize to another's); the 35B FAIL is the guardrail on that tradeoff; the 50% aggressive-ratio cliff deliberately not crossed; vision-conditioned quality open; trainability unproven pending the gate. | The limits are the design's own shape (**the central tradeoff**): pruning to one operator's work means the result is that operator's model, not everyone's. The failed replication is kept as the boundary marker, the deeper-cut regime was deliberately not attempted, and the two open questions stay labeled open. | #### 11. Conclusion **The point:** the close in the chapter's own line: sovereignty is not only where the weights live but whose distribution shapes them. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A 122B MoE shipped for datacenter service now runs for a single operator on a 12 GB card, pruned to the experts that operator's workload uses, quantized under a matrix calibrated on that operator's corpus, at higher fidelity than the quantization-only alternative at equal-or-better speed; the failed replication is reported beside the success because a compression method is only as trustworthy as its worst honestly-measured case. | The summary earns its headline (**the flagship result**) and states its own credibility rule: the failure is printed next to the success because that is what makes the success believable. | | The through-line to the series: sovereignty is not only a matter of where the weights live but of whose distribution shapes them; the operator's own corpus is the instrument that decides, at the architecture layer as at the quantization layer, what to keep. | The closing thesis (**whose distribution shapes them**): owning the machine is half of sovereignty. The other half is that your own work, not someone's generic sample, decides what the model keeps. | #### System Update: July 2026 **The point:** the append-only update: the replacement held, tuning found headroom, and the vision risk is exercised but not closed. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The serving preset was repointed from the full 2-bit build to the pruned 4-bit build on 2026-07-06, the day the gate sealed, and has served continuously since; the superseded entry is commented out rather than deleted, so the substitution is legible in the configuration; nineteen days of residency at writing, an availability record, not a quality claim. | The decision was executed, not just recommended (**the replacement held**): the pruned model took the production seat the day its gate sealed, and the old configuration line is kept visible, struck through, as its own receipt. | | An operational note records \~18.5 tok/s at a lower expert-offload setting within the same memory envelope, above the gate's 16.65, with a spill cliff one step further down; reported as a deployment configuration note, not a sealed measurement, the sealed comparison unchanged. | Tuning later found free speed (**headroom in the offload boundary**), reported with its label attached: a settings note, not a sealed result, and one step past it the familiar cliff still waits. | | A second preset serves the same pruned weights with the multimodal projector in a finished-video inspection role: production exercise of the §7.2 risk without spill or rollback, offered explicitly as the weaker evidence: deployment without incident is not a passed gate, and §7.2 stays open exactly as written. | The vision question got real-world exercise (**exercised, not closed**): the pruned model now inspects video frames in production without incident, and the chapter still refuses to count that as proof. Running clean is weaker evidence than passing a gate, and it is offered only as the weaker thing. |