> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/part-iv-the-evidence-what-worked/15-sovereign-imatrix-calibration/the-quick-version.md). # The quick version **Continue the tour →** [Next: 16 · Sovereign Domain Pruning, the quick version](/osintelligence/part-iv-the-evidence-what-worked/16-sovereign-domain-pruning/the-quick-version.md) The short version of Chapter 15, three ways: the video walks the argument in a few minutes, the deep dive talks it through at a listening pace, and the infographic holds the whole chapter in one view. The full result, with the 16× tighter divergence and the supply-chain reframing where the defense is also the quality win, lives in the chapter itself: [15 · Sovereign Imatrix Calibration](/osintelligence/part-iv-the-evidence-what-worked/15-sovereign-imatrix-calibration.md). {% embed url="" %} **The deep dive.** A podcast-style audio conversation about this chapter: two AI hosts walk through the argument, the incidents behind it, and what it means, at a listening pace. Generated in Google's Gemini LM (formerly NotebookLM) from the chapter itself; the link opens the audio on Google's site. {% embed url="\_" %} *The conversation is AI-generated: an interpretation of the chapter, not the chapter. It can compress, paraphrase, or get details wrong. The written chapter is the authoritative, canonical source:* [*15 · Sovereign Imatrix Calibration*](/osintelligence/part-iv-the-evidence-what-worked/15-sovereign-imatrix-calibration.md)*.* *** ![The Sovereign Imatrix Calibration, the chapter in one view.](https://137900913-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fxx2bv6VR9dSJDJ9HsDER%2Fuploads%2Fv6hS9d7DzVreo8kzL41L%2Fimatrix-infographic.png?alt=media) *** ### Chapter notes Section-by-section notes in two registers: the technical note on the left, the same idea in plain language on the right. Every row is one idea, so you can read straight across from one register to the other. The technical terms stay visible in the plain column on purpose; they are the vocabulary worth keeping. #### Abstract **The point:** one substitution, calibrate the compression on the operator's own data instead of generic text, and the result is smaller, faster, and 16× more faithful at once. | The technical note | In plain language | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Substituting a workload-specific in-distribution corpus (511 operator memory breadcrumbs plus targeted dog-food synthesis) for a generic one during mixed-precision quantization of a 35B-A3B MoE yields mean KL divergence 0.0103 against BF16, versus the published community Q6\_K band of \~0.16–0.20: a 16× improvement at equivalent compute cost. | Shrinking a model for consumer hardware (**quantization**) involves a tuning step almost everyone feeds with generic internet text (**the calibration corpus**). Feeding it the operator's own working data instead made the shrunken model 16× more faithful to the original (**KL divergence**: how far the compressed model's answers drift from the uncompressed one's). | | The artifact simultaneously shrinks 21% (22.8 vs 28.9 GB), preserves 96.08% top-1 token agreement, holds perplexity within ±0.11% on both in-distribution and out-of-distribution sets, matches the baseline 1:1 on a deterministic code audit, and runs 1.11× faster. | And nothing was traded away for it: the file got a fifth smaller, the model answers the same, passes the same audit, and runs 11% faster. The three things that normally trade against each other all improved together. | | The load-bearing reframe: calibration is a training-time input with zero inference-time footprint, which makes it a supply-chain attack surface; the same 16× that rewards the curating defender is available in reverse to an attacker who mis-curates a corpus that still scores clean on wikitext. | The real payload is the security reframe (**calibration as attack surface**): this tuning data vanishes after the build, invisible to anyone downstream. The same lever that gave the defender 16× could give an attacker the reverse, undetectably. Curating your own is both the quality win and the defense. | #### 1. Introduction **The point:** the community ships quantizations tuned on generic prose to operators with specific workloads; the paper asks the obvious unasked question. | The technical note | In plain language | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GGUF imatrix calibration collects activation statistics over a calibration corpus to select per-tensor quantization parameters; community practice (Unsloth, bartowski, individual quantizers) treats the corpus choice as secondary and uses generic mixes. | The mainstream toolchain already supports workload-aware compression (**the imatrix**); the community just feeds it generic text and moves on. Millions of downloaded models are tuned for an average workload nobody actually runs. | | The question: what if we calibrate on our actual workload? The subject is a 35B mixture-of-experts (40 layers, 256 experts, 8 active per token) quantized from BF16 with an imatrix computed over the operator's own memory store and dog-food synthesis. | The experiment is one honest sentence (**what if we calibrate on our actual workload?**), run on the stack's big architect model, using text harvested from the operator's own working memory. | | The finding stated up front: not that the sovereign quantization matches its baseline, but that it exceeds published community benchmarks by an order of magnitude on the standard fidelity metric while shrinking the artifact and accelerating inference; the paper is the receipt for that sentence. | The claim is bigger than parity (**the receipt sentence**): the home-tuned version did not just keep up with standard practice, it beat the published field by an order of magnitude while getting smaller and faster. The rest of the chapter is the paperwork behind that one sentence. | #### 2. Background **The point:** how the mechanism works, why the corpus question was never really asked, and why KL divergence is the honest ruler. | The technical note | In plain language | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | llama-imatrix runs the reference model over the calibration corpus and records per-tensor activation statistics; llama-quantize consumes them so per-tensor rounding minimizes squared-activation-weighted error over the calibration distribution. | The mechanism in one line (**importance-weighted rounding**): the tool watches which parts of the model the sample text actually exercises, then spends its compression precision where the traffic is. Which is exactly why the choice of sample text matters. | | Community corpora (wikitext, C4 shards, curated mixes) are tuned for generality; none represents the token distribution a given operator's deployment actually sees; a model calibrated on news prose gets evaluated against threat-intel classification. | The standard sample texts are chosen to be average on purpose (**the corpus question**). Average is precisely wrong for any specific operator: a model tuned on news articles is being asked to be precise about threat intelligence. | | KL divergence measures distributional similarity across the full vocabulary against the BF16 reference, unlike perplexity, which token-level agreement can game; published Q6\_K-tier benchmarks sit at \~0.16–0.20 mean KLD, Q5 at 0.40–0.50, Q4 at 0.60–0.80. | The ruler is chosen carefully (**KL divergence over perplexity**): it compares the whole probability distribution against the uncompressed original, not just whether the top answer matches. The published field's typical scores set the bar the result is measured against. | #### 3. Method **The point:** the corpus recipe (harvest, dog-food, iterate), the mixed-precision schedule, and four pass/fail gates fixed before the build. | The technical note | In plain language | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Phase A harvests the operator's Hindsight memory store across three banks (six recall queries each, ≥80-char filter, ground-truth bonus): 511 breadcrumbs, \~230K characters of real operator/agent interaction; Phase B prompts the target model itself across representative task types (code review, doctrine synthesis, intelligence analysis) to fill the reasoning-density gap; Phase C iterates six corpus revisions, each checked by a fast-fail KLD probe, v6 first to stabilize below 0.05. | The recipe has three phases (**harvest, dog-food, iterate**): pull real working text from the system's own memory, have the model itself write samples of its hardest task types (**dog-food synthesis**), and refine the blend six times with a quick fidelity probe until it stabilizes. Nothing exotic: the data was already on the disk. | | The APEX I-Balanced schedule places attention projections, router gates, shared-expert FFN, and embeddings at Q6\_K while the bulk expert FFN goes to IQ4\_XS with imatrix; target ≤23.6 GB, actual 22.8 GB. | The compression is not uniform (**mixed precision**): the model's steering machinery keeps high precision while the bulk gets compressed harder, guided by the calibration. That is where the size win comes from without a quality cost. | | Four validation gates were defined pre-quantization: V1 perplexity deltas on sovereign and wikitext-2 held-outs, V2 full-vocab KLD (mean ≤0.30, max ≤5.0), V3 a 15-prompt deterministic SAT audit judged by an independent Orchestrator, V4 VRAM peak and throughput ratio; all on the same RTX 5070 substrate. | Four pass/fail tests were locked before the build ran (**the pre-registered gates**): quality on both home data and neutral data, distribution fidelity, a code-audit face-off judged by a third model, and the hardware budget. No gate could be invented after seeing results. | #### 4. Results **The point:** every gate green with margin: fidelity 29× inside its own acceptance bar, parity everywhere quality was measured, and faster on top. | The technical note | In plain language | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | All gates pass: sovereign PPL −0.082% (bar ±1.0%), wikitext-2 −0.107% (bar ±3.0%), mean KLD 0.0103 (bar ≤0.30), max KLD 1.52 (bar ≤5.0), top-1 agreement 96.08%, SAT audit 53.3% APPROVE both arms, peak VRAM 11.2 GB (bar ≤23.6), throughput 1.11× (bar ≥0.90×). | The scoreboard is uniformly green with room to spare (**all gates green**): quality identical on both test sets, fidelity thirty times inside its own acceptance bar, the code audit a 1:1 tie, and the hardware numbers better than required. | | The KLD detail: mean 0.010327 ± 0.000293, median 0.004351, 99th percentile 0.093, maximum 1.52; the distribution sits an order of magnitude left of the published community Q6\_K band (Figure 1). | The fidelity result is not a lucky average (**the distribution**): even the 99th-percentile worst tokens drift less than the field's typical mean. The whole curve sits an order of magnitude better, not just its midpoint. | | Perplexity parity on both the 10 MB sovereign held-out and wikitext-2, with APEX fractionally (not significantly) better on both, at \~1.3× the baseline's evaluation throughput; the out-of-distribution parity is the evidence fidelity was not bought by overfitting the calibration domain. | The crucial honesty check (**the out-of-distribution test**): quality on neutral internet text also held exactly, which is the evidence the home-tuning did not just memorize home data at the expense of everything else. | | The SAT audit ran 15 coding prompts through both models under an independent 9B judge on a fixed rubric: APPROVE 8/8, FLAG 7/7, REJECT 0/0, a 1:1 match; V4 measured 324 MB less peak VRAM, 11% higher throughput, 6.1 GB saved on disk. | A third model refereed a coding face-off and could not tell the two apart (**the 1:1 audit**). Meanwhile the compressed version uses less memory, less disk, and answers faster. | #### 5. Discussion **The point:** why it works, the honest limits, and the chapter's real payload: the calibration file as a supply-chain surface, with this method as the defense. | The technical note | In plain language | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Attribution: corpus-workload alignment; Phase A captured the exact interaction patterns the model is evaluated against, Phase B the reasoning density it must produce; community corpora trade calibration fidelity on any specific workload for generality; the 96% top-1 figure is the separate autoregression-relevant metric, consistent with MoE expert-selection non-determinism; the speed win is bandwidth: 21% less footprint on a 504 GB/s card. | The mechanism explained (**corpus-workload alignment**): precision went exactly where this operator's traffic goes. The remaining 4% top-answer disagreement is expected noise in a mixture-of-experts, and the speed win is simple physics: fewer gigabytes to move per token. | | §5.4 carries the toolchain disclosure: all numbers were measured on the CUDA 13.1 binary; the companion paper's later investigation prompted a 12.8 rebuild, with Gate-B parity holding PPL within ±0.5% and throughput within ±5%; future re-runs should expect headline deltas to shift by at most that band. | A reproducibility footnote is volunteered rather than buried (**the toolchain disclosure**): the serving binary was later rebuilt for reasons the next chapter documents, the numbers were re-checked on the new build, and the expected wobble band is stated for anyone reproducing. | | §5.5's limits: single model, single baseline comparator (published numbers from a different model's card, not a same-model wikitext-imatrix head-to-head), the calibration-leakage concern (dog-food text comes from the model being quantized; the clean OOD delta argues against the pathology but does not fully close it), and no speculative-decoding benchmark yet. | The honesty ledger names the four soft spots (**the limits**), including the two a critic would reach for first: the comparison uses the field's published numbers rather than a same-model rematch, and the model helped write its own tuning text, a loop the neutral-text result argues against but does not fully close. | | §5.7 composes the threat surface in three observations: calibration data never appears at inference time (unrecoverable from the weights), distribution-specific drift is invisible under standard evaluation (clean on wikitext, failing selectively on the drifted distribution), and the attack generalizes to every training-time ingredient with no inference-time signature (silver labels, distillation corpora, preference pairs); the countermeasure is sovereign in-distribution calibration, dual-signaled as quality win and supply-chain property, operationalizable by changing only the imatrix input. | The payload argument (**the supply-chain reframe**): the tuning data is burned after the build, so a poisoned batch is undetectable downstream and invisible to every standard benchmark. The same trick works on any training ingredient that leaves no runtime trace. The defense is the method itself: curate your own, and the audit trail and the quality gain arrive together (**dual-signaled**). | #### 6. Conclusion **The point:** one strong data point for a general thesis: the corpus was already on the disk, the lever was already in the toolchain, and the measurement shows what the combination is worth. | The technical note | In plain language | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The quantization is simultaneously smaller (−21%), faster (+11%), and 16× more faithful than the community baseline; the thesis: corpus-workload alignment is a first-order lever in post-training quantization, routinely underexploited; if the effect replicates, the default advice changes from "pick the model that matches your workload" to "calibrate whatever model you pick to the workload itself." | The close is measured (**a single strong data point**): one model, one operator, all gates green, and a thesis worth testing everywhere: do not just pick a model for your work, tune its compression to your work. | | The paper's framing of its own contribution: the corpus was already on the operator's disk, the lever was already in the toolchain; what the paper adds is the measurement showing how much the combination is worth. | The final sentence keeps the discovery humble (**the lever was already there**): nothing new was invented. Something free was measured, and it turned out to be worth an order of magnitude. | #### System Update: July 2026 **The point:** the append-only update: the corpus became a governed artifact, the discipline spread to every compression layer, and the quantization went to production. | The technical note | In plain language | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The calibration corpus matured from a one-off harvest into a governed artifact: every run recomputes a SHA-256 sidecar that hard-fails on drift (negative-tested: one flipped byte aborts the build), the slice now renders from tens of thousands of governed rows with per-source provenance, and the method generalized to a governance-classifier adapter that cleared all four fidelity gates. | The tuning data grew up (**a governed artifact**): it is now fingerprinted before every build, a single tampered byte stops the machine (and that guard was tested by tampering on purpose), and the recipe has already been reused on a second model, passing the same four gates. | | A corpus-identity discipline now binds one sealed sovereign corpus draw to every corpus-touching layer (prune-saliency, recovery fine-tuning, imatrix, self-distillation, persona adapters, the draft head), making the compounding question testable as a pre-registered matched-versus-mismatched study with the mismatched arm frozen as control; the compression flagship inherits the discipline. | The bigger echo (**one corpus, every layer**): the whole compression pipeline now cites a single fingerprinted data draw, which turns this chapter's one-layer result into a falsifiable whole-chain question, already pre-registered with its control arm frozen. | | The APEX artifact was promoted to the operational architect model on 2026-06-09, serving every stage of the production broadcast pipeline under per-inference telemetry, turning the §4 throughput and quality-equivalence verdicts into continuously monitored production quantities; stated honestly as observational validation, not a controlled re-run, with the stock architect retained for A/B fallback. | And the artifact itself went to work (**promoted to production**): the model this chapter validated now runs the daily broadcast under live monitoring, with the honest label attached: that is production evidence, not a re-run of the sealed gates, and the original model stays registered as the fallback. |