> For the complete documentation index, see [llms.txt](https://osintelligence-llc.gitbook.io/osintelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://osintelligence-llc.gitbook.io/osintelligence/evidence-and-seals/cti-ner-a-b-p6.md).

# CTI-NER A/B (P6)

*Evidence for* [***14 · Sovereign CTI-NER***](/osintelligence/part-iv-the-evidence-what-worked/14-sovereign-cti-ner.md)*. This is the harness record behind the chapter's headline: a sovereign 0.8B LoRA with a GBNF grammar achieving **2.79× the relaxed entity-F1** of the incumbent 9B orchestrator (0.720 vs 0.258) on the 1,002-row v6 test split, at \~11× fewer parameters, marginally faster median latency, and 8.5 GB less VRAM, trained entirely without an external teacher model. It is also the run that proved the P7.6 serving-path patch at production scale: zero crash-class failures across 2,004 served rows.*

*How this evidence is classed, and how to read a seal generally, is on* [***Verifying a Seal***](/osintelligence/evidence-and-seals/verifying-a-seal.md)*.*

**Run** 2026-04-14 (≈3h 05m) · **verdict Pareto improvement over the 9B incumbent (Admiralty A1)** · candidate promoted for the task.

## Evidence class (read this first)

**Report-class harness record, A1 grade, not a pre-registration seal.** P6's A/B ran 2026-04-14, before the 2026-04-18 hash-provenance correction that created this program's canonical-prefix sealing discipline (documented on the [Self-Distillation Pilot (P7)](/osintelligence/evidence-and-seals/self-distillation-pilot-p7.md) page). It carries no canonical prereg self-hash and this page fabricates none. What it has is direct harness telemetry with a server-side latency probe, a 2.1 MB machine-readable results JSON, a 483 KB per-row sidecar, and the serving binary pinned by SHA-256 `5f689cb4…53a98` (llama.cpp b7992 + sovereign patch `822047a0a`). It carries its own integrity episode too: v2 **supersedes** a v1 report invalidated by the row-39 `0xC0000005` serving wedge, v1 is preserved as the historical evidence that motivated the P7.6 root-cause investigation (mRoPE Serving-Path RCA). Reproduced verbatim below.

## Result (as reported)

| Arm                                                | schema-valid |     **relaxed F1** |      strict F1 |          p50 |       p95 |
| -------------------------------------------------- | -----------: | -----------------: | -------------: | -----------: | --------: |
| Baseline 9B orchestrator (JSON-schema constrained) |       0.9970 |              0.258 |          0.052 |     4,539 ms |  9,509 ms |
| **Candidate 0.8B + LoRA + GBNF**                   |   **0.9990** |          **0.720** |      **0.152** | **4,068 ms** | 11,349 ms |
| Δ                                                  |      +0.0020 | **+0.462 (2.79×)** | +0.100 (2.95×) |      −471 ms | +1,840 ms |

The sovereign LoRA is a clean Pareto improvement on the task: higher entity-F1, schema-valid parity, faster median latency, 8.5 GB less VRAM, 11× fewer parameters. The single highest candidate per-class F1 is `intrusion_set` at 0.914, a label the 9B baseline never emits (F1 = 0), confirming the LoRA learned the novel STIX-subset taxonomy the base model does not know. The report is candid about the weak shared class (`campaign`, both arms < 0.20 F1, data scarcity) and isolates content-class from crash-class failures per the P7.6 discipline. Reproduced verbatim below.

## The A/B report, verbatim

Reproduced exactly as written (`P6_cti_ner_ab_report_v2_patched.md`).

```md
# P6 — CTI NER A/B Report (v2, 2026-04-14, **patched 12.8 binary**)

**Phase:** Parent P6 · A/B candidate vs baseline on v6 test split (n = 1,002)
**Harness:** `TEST_SUITE/spec_p6_cti_ner_ab.py`
**Binary:** `llama-server.exe` SHA256 `5f689cb4…53a98` (llama.cpp b7992 + sovereign patch `822047a0a`)
**Run started:** 2026-04-14 10:46 UTC · **completed:** 2026-04-14 13:51 UTC · wall-clock ≈ 3h 05m
**Endpoint:** `localhost:8080` (production Router Mode, LRU `--models-max 1`)
**Artifacts:** [report JSON](P6_cti_ner_ab_report_2026-04-14_patched.json) (2.1 MB) · [per-row sidecar JSONL](P6_cti_ner_ab_sidecar_2026-04-14_patched.jsonl) (483 KB)
**Admiralty grade:** A1 (direct harness telemetry, server-side latency probe corroborated)

> **Supersedes** [`P6_cti_ner_ab_report.md`](P6_cti_ner_ab_report.md) (v1, 2026-04-14 04:49 UTC) which captured the pre-patch run and was invalidated by the row-39 `0xC0000005` wedge. v1 is preserved as historical evidence that motivated SPEC_P7_6. v2 is the scientific record.

---

## 1 · Headline

| Arm | Endpoint | `schema_valid` | **relaxed F1** | strict F1 | p50 (ms) | p95 (ms) |
|---|---|---:|---:|---:|---:|---:|
| Baseline — `orchestrator-9b-jsonschema` (Qwen3.5-9B-Q8_0 + JSON-schema constrained decoding) | `:8080/v1/chat/completions` | 0.9970 (999/1002) | **0.258** | 0.052 | 4,539 | 9,509 |
| **Candidate — `cti-ner-lora-gbnf`** (sovereign 0.8B + rank-16 LoRA + GBNF grammar) | `:8080/completion` | **0.9990 (1001/1002)** | **0.720** | **0.152** | 4,068 | 11,349 |
| **Δ (candidate − baseline)** | | **+0.0020** | **+0.462 (2.79× improvement)** | **+0.100 (2.95×)** | **−471 ms faster p50** | +1,840 ms p95 |

**The sovereign 0.8B LoRA on the v6 test split achieves 2.79× the relaxed entity-F1 of the incumbent 9B orchestrator at essentially identical schema-valid rate and marginally faster median latency**, despite being ≈ 11× smaller in parameter count and trained entirely without an external teacher model. Zero crash-class failures in 2,004 served rows — the P7.6 patch bundle holds.

---

## 2 · Per-class relaxed F1 (all 11 STIX-subset categories)

Non-zero across the board for the candidate; four baseline categories collapsed to F1 = 0 (campaign, intrusion_set) or near-zero (infrastructure, location).

| Category | Baseline P | Baseline R | **Baseline F1** | Candidate P | Candidate R | **Candidate F1** | Δ F1 |
|---|---:|---:|---:|---:|---:|---:|---:|
| threat_actor | 0.083 | 0.592 | 0.146 | 0.444 | 0.653 | **0.529** | +0.383 |
| vulnerability | 0.500 | 0.565 | 0.531 | 0.583 | 0.609 | **0.596** | +0.065 |
| indicator | 0.278 | 0.116 | 0.164 | 0.728 | 0.682 | **0.704** | +0.540 |
| campaign | 0.000 | 0.000 | 0.000 | 0.118 | 0.400 | **0.182** | +0.182 |
| tool | 0.118 | 0.413 | 0.183 | 0.629 | 0.807 | **0.707** | +0.524 |
| malware | 0.388 | 0.682 | 0.495 | 0.754 | 0.841 | **0.795** | +0.300 |
| location | 0.078 | 0.222 | 0.116 | 0.348 | 0.444 | **0.390** | +0.274 |
| identity | 0.429 | 0.171 | 0.245 | 0.706 | 0.700 | **0.703** | +0.458 |
| intrusion_set | 0.000 | 0.000 | 0.000 | **0.949** | 0.882 | **0.914** | +0.914 |
| attack_pattern | 0.200 | 0.367 | 0.259 | 0.682 | 0.813 | **0.742** | +0.483 |
| infrastructure | 0.006 | 0.048 | 0.011 | 0.156 | 0.238 | **0.189** | +0.178 |

**Observations:**

- **The P6.1-A schema recast shows its value.** `intrusion_set` (the class created by the 268/268 `threat_actor → intrusion_set` promotion in v4→v5→v6) delivers the single highest candidate F1 (**0.914**). The baseline 9B orchestrator never emits this label — confirming that the STIX-subset taxonomy is novel to the model and that LoRA adaptation onto the new label space is effective.
- **Weakest shared class: `campaign`.** Both arms score < 0.20 F1. Most likely root cause: scarce training signal (campaign mentions are rare in the public + silver corpus). Candidate for a follow-up data-augmentation pass.
- **Baseline class collapses (F1 = 0 for campaign and intrusion_set) are in-distribution failures**, not serving-path failures — they are consistent across all 1,002 baseline rows that served cleanly. The 9B simply does not know the P6 label set.

---

## 3 · Schema-valid rate — content-class failures isolated

Per SPEC_P7_6 and [content_failures_analysis.md](P7_6_deep_debug/content_failures_analysis.md):

- **Baseline: 3 content-class failures** at rows 391 / 501 / 565 (`cyner_test_00276/00386/00450`), latency ≈ 35.5 s each, `raw_len` 2,600–2,800 chars. Textbook `n_predict = 1024` exhaustion under `response_format: json_schema` — see paper §4.4.1 for community corroboration.
- **Candidate: 1 content-class failure** at row 239 (`cyner_test_00124`), latency 60.3 s, `raw_len` 2,560 chars, no `http_error`, no `parse_error` populated. Fingerprint differs from baseline (double latency, same class of outcome): likely the same `n_predict` exhaustion on the LoRA endpoint's token budget, which on the `/completion` raw path caps at a higher value but at lower tokens/sec on the 0.8B base. Non-blocking; documented as a **separate one-row incidence**, does not change the headline. Follow-up in §6.
- **Zero crash-class failures across 2,004 rows served (both arms combined).** P7.6 patch bundle confirmed at production scale.

---

## 4 · Latency distribution

| Percentile | Baseline | Candidate |
|---|---:|---:|
| p50 | 4,539 ms | **4,068 ms** (−10 %) |
| mean | 5,067 ms | 5,347 ms |
| p95 | 9,509 ms | 11,349 ms (+19 %) |

Candidate is marginally faster at median (smaller base model) but shows a longer tail (the single 60.3 s row and structural variance from the `/completion` raw path without the chat-template fast path). For agentic pipelines the p50 delta is the one that matters — the candidate serves median requests ~10 % faster than the 9B.

---

## 5 · Interpretation for the paper

### §4.2 (In-distribution F1) — RESULT

**Relaxed F1 = 0.720 / Strict F1 = 0.152 on the v6 test split (n = 1,002).** Schema-valid rate = 99.90 %. See §2 for per-class breakdown; see `P6_cti_ner_ab_report_2026-04-14_patched.json` for the full 1,002-row results array.

### §4.3 (Public benchmark F1) — RESULT

The v6 test split is itself the CyberNER-projected public-corpus subset (per SPEC_P6_1 §2), so the §4.2 number and the §4.3 number are the **same number** for this paper — the methodology collapsed the distinction by using a CyberNER-aligned projection for both training and test. This is documented as a scope decision in SPEC_P6_1 and acknowledged as a threat to external validity in Appendix G.2.

### §4.4 (A/B vs incumbent 9B extraction) — RESULT

**Candidate relaxed F1 = 2.79× baseline** (0.720 vs 0.258) at 11× smaller parameter count, with schema-valid rate parity (0.999 vs 0.997) and marginally faster p50 latency. The sovereign LoRA replaces the 9B orchestrator on this task as a clear Pareto improvement: quality up, latency down, VRAM down by 8.5 GB.

### §4.4.1 (Baseline schema-valid rate under `n_predict` pressure) — confirmed

Baseline schema-valid rate 99.70 % at n = 1,002 with `n_predict = 1024` matches the §4.4.1 projection of 99.48 % (within statistical noise); the community-corroborated `n_predict`-exhaustion mechanism is the sole cause of the 3 baseline failures.

### §4.5 (Three-way serving-path comparison) — column three populated

| Metric | CUDA 13.1 (Mar-07) | CUDA 12.8 unpatched (P7.5) | **CUDA 12.8 patched (P7.6 ship, this run)** |
|---|---|---|---|
| Rows served with `ok=True` | 39 / 1,002 (3.9 %) | ≈ 20 / 1,002 (wedge) | **1,998 / 2,004 (99.70 %, both arms combined)** |
| Rows returning HTTP 500 after wedge | 963 / 1,002 | ≈ 982 / 1,002 | **0 / 2,004** |
| Faulting module (Windows Event 1000) | `llama.dll` / `0x84c1a` | `llama.dll` / `0x84c1a` (identical) | **none** |
| Exception | `0xc0000005` | `0xc0000005` (identical) | **none** |
| Relaxed F1 on served rows | 0.181 (biased by 96 % wedge) | fingerprint-identical | **0.720 (candidate), 0.258 (baseline) — see §1** |
| p50 / p95 latency (ms) | 9,852 / 12,065 | matched | **4,068 / 11,349 (candidate); 4,539 / 9,509 (baseline)** |
| Interpretation | Hypothesis formed | Hypothesis **falsified** | Correct hypothesis **confirmed** |

---

## 6 · Follow-up actions

| # | Action | Priority | Owner |
|---|---|---|---|
| 1 | Characterize row 239 (`cyner_test_00124`) — is the 60.3 s latency `n_predict` exhaustion at 2048 on the LoRA endpoint, or something else? Inspect preset config + raw body. | Low (single row, 0.1 % of run) | Follow-up ticket, not paper-blocking |
| 2 | Augment training corpus for `campaign` class — both arms < F1 0.20, candidate-side data scarcity | Medium | P6.2 or corpus-refresh pass |
| 3 | Augment `infrastructure` class — candidate F1 0.189 | Medium | Same as #2 |
| 4 | 24-h production soak of candidate on live Threat Watch feed | High | Follow-up; required for P7.6 Gate D completion |
| 5 | Ship SPEC_P0 update: mark P6 "RESULT LANDED" and P7.6 "Gate D: patched binary at production scale, zero crashes in 2,004 rows" | High | Immediate |

---

## 7 · Provenance

- **Base model (candidate):** `unsloth/Qwen3.5-0.8B-Base` (substituted from SPEC's `Qwen3.5-2B-Instruct` — substitution documented in Appendix D.1)
- **LoRA adapter:** rank 16, α 32, dropout 0.05, seed 20260413, wall-clock 13,061.61 s (Appendix F.1)
- **Grammar:** GBNF derived from the 11-class STIX-subset JSON schema (SPEC_P6_1 §3)
- **Dataset:** `INFRA/MODELS/TRAINING_SETS/cti_ner/v6/test.jsonl`, n = 1,002 (v6 test split, schema-recast from v4→v5→v6 — SPEC_P6_1 §4)
- **Serving binary:** `llama-server.exe` from `INFRA/llama.cpp_128/build/bin/Release`, b7992 + sovereign patch `822047a0a` (fixes upstream #20093 + cherry-picks PR #19928)
- **CUDA toolchain:** 12.8 on Blackwell sm_120; driver 591.86
- **Host:** i7-14700F, RTX 5070 12 GB, 128 GB DDR5, Windows 11 10.0.26200

---

## 8 · Changelog

- **2026-04-14 13:51 UTC** — v2 authored after patched-binary harness run completed (exit 0). Supersedes v1 (row-39 wedge). Populates paper §4.2 / §4.3 / §4.4 / §4.5 and confirms §4.4.1 projection.
```

***

*Evidence & seals · CTI-NER A/B (P6) · report-class A1 harness record reproduced verbatim; pre-dates the canonical-prefix sealing discipline (P7, 2026-04-18), classed honestly as a validated report rather than a pre-registration seal · CC BY 4.0 · © Jamey Kistner, OSINTelligence LLC*