---
project: modelmap
document: expA_probe_reliability — analysis (run #1)
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Analysis — expA run #1: the harness failed its own validity gate (and that is the result)

Run: `results/expA_probe_reliability/20260812T062605Z/results.json` (48 s wall).
Model: mlx-community/Qwen3-0.6B-4bit (28 layers) + random-init architecture twin.
Grid: 3 properties × 2 disjoint promptsets × 28 layers × 5 seeds, shuffled-label
control inside every probe, BH-FDR q=0.05, top-5 replication. Hypothesis and
validity criterion registered before the run.

```text
Hypothesis              : probe maps are seed-stable but dataset-sensitive
                          (dataset variance > seed variance on ≥1/3 of layers).
Falsification criterion : seeds dominate; AND harness validity gate: twin
                          (random-init) selectivity must stay < 0.05.
Result                  : PRIMARY HYPOTHESIS UNTESTABLE — ceiling effect:
                          task accuracy 1.000 at every layer, every property,
                          both sets; seed SD = 0.0000; dataset shift = 0.0000.
                          VALIDITY GATE FAILED, decisively: the twin reaches
                          accuracy 1.00 at all 28 layers for all properties
                          (twin max selectivity 0.56–0.88 vs the registered
                          0.05 bar); mean real-minus-twin selectivity is
                          within ±0.06 — i.e. ZERO measurable trained-model
                          signal in the map. 28/28 layers "FDR-significant"
                          in the twin too: the statistics are fine, the
                          measurand is wrong.
Interpretation          : Level 1 FOR THE NEGATIVE CLAIM (controlled,
                          replicated across 5 seeds × 2 sets × 3 properties):
                          on lexically separable classes, per-layer linear
                          probes on mean-pooled representations measure the
                          tokenizer + architecture prior, not learned
                          computation. This is the probing-pitfalls
                          literature (notes §4.1/§4.8 — Hewitt & Liang,
                          Bolukbasi, Dead Salmons) reproduced in our own
                          hands on our own harness, caught by a
                          pre-registered gate rather than by a reviewer.
                          Published as the atlas's first entry
                          (atlas/qwen3-0.6b-4bit/probes/v1,
                          negative_result=true) — maps that fail their nulls
                          are first-class citizens here.
Next experiment         : expA run #2 with (a) promptsets v2 that remove
                          lexical separability (shared vocabulary across
                          classes; structure-borne properties), (b) probe
                          maps published only as REAL−TWIN differentials,
                          (c) last-token representations at matched
                          positions alongside mean-pooling, (d) a
                          class-balanced token-overlap statistic reported
                          per promptset as a difficulty certificate.
```

## Notes

- The ceiling itself was foreseeable in hindsight: template-generated classes
  differ lexically (FR vs EN vocabulary; `def`/`const` tokens; digit tokens),
  and random features over distinct token distributions are linearly
  separable. v1 promptsets keep their declared role — they are now the
  *positive control* corpus (any harness that fails to reach ceiling on them
  is broken) rather than a measurement corpus.
- The twin null earns permanent-fixture status: it is cheap (one extra
  capture pass) and it is the only control in the battery that caught this.
  Shuffled-label controls passed everywhere (selectivity 0.3–0.7) and would
  have let the illusion through.
- Wall-clock: full grid in 48 s on M5 Max — noise-floor science at this scale
  is essentially free; the expensive part was thinking, not compute.

---

# Analysis — expA run #2: differential maps work; the twin keeps teaching

Run: `results/expA_probe_reliability/20260812T063856Z/results.json` (88.6 s).
v2 promptsets (structure-borne, token-balanced; overlap certificates in the
manifest), mean-pooled AND last-token reps, twin null on A sets, v1
positive control. Hypothesis registered before the run.

```text
Hypothesis (a) VALIDITY : twin max selectivity < 0.05 on token-balanced sets.
Result (a)              : FALSIFIED AGAIN — but the gradient is the finding.
                          word_order: twin acc 0.958–0.963 (sel 0.47–0.59);
                          agreement: twin acc 0.683–0.729 (sel 0.15–0.23);
                          arith_valid: twin acc 0.558–0.579 (sel 0.08–0.12).
                          Even with matched word sets, TOKENIZATION statistics
                          differ between classes (mid-sentence capitals,
                          position-dependent subwords in scrambles; is/are
                          bigram contexts) — the twin mines them. The strict
                          0.05 gate is unreachable for surface-correlated
                          properties; the differential doctrine (v1) is the
                          correct instrument, and v2 applies it.
Hypothesis (b) SIGNAL   : real−twin sel > 0.10 on ≥5 FDR-significant layers
                          for word_order AND agreement.
Result (b)              : SPLIT. agreement PASSES decisively — 25/28 layers
                          (mean pooling; 23/28 last-token), max Δsel +0.379,
                          real acc up to 0.967 vs twin 0.729. word_order
                          FAILS — 2/28 layers (0 last-token), max Δsel
                          +0.129: grammatical-vs-scrambled is essentially
                          null-dominated; NOT valid evidence of learned
                          syntax at this granularity.
Hypothesis (c) arith    : exploratory.
Result (c)              : POSITIVE SIGNAL — real acc 0.858 (mean) / 0.904
                          (last-token) vs twin 0.558/0.579; 5–6 signal
                          layers; max Δsel +0.358. Equation-validity is
                          linearly decodable above the architecture prior in
                          a 0.6B-4bit model, strongest in last-token reps
                          (consistent with computation completing at "=").
Hypothesis (d) variance : dataset shift > seed SD on ≥1/3 of layers.
Result (d)              : NOT SUPPORTED — 3–8/28 layers across cells. Off
                          ceiling, seed SD is finally nonzero and top-5-layer
                          replication drops to ~0.54 (vs the trivial 1.00 at
                          ceiling): the FIRST real noise-floor numbers of the
                          project. Seed and dataset variance are comparable
                          in this regime; neither dominates.
Positive control        : v1 lang_id_A at ceiling (1.000) — harness intact.
Interpretation          : Level 1 per property, differential claims only.
                          agreement + arith_valid are the project's first
                          POSITIVE maps (published in atlas probes/v2 with
                          per-property verdicts); word_order is flagged
                          null-dominated in the same entry. Twin gate:
                          retired as a binary gate, kept as a mandatory
                          reported baseline — the differential IS the map.
Next experiment         : (1) promote agreement/arith_valid toward Level 2:
                          second method (LEACE erasure damage) on the same
                          sets; (2) expC causal check on the top agreement
                          layers (ablation); (3) noise-floor deepening: more
                          seeds at fixed set to tighten replication CIs
                          (expD); (4) quantization drift of the agreement
                          differential map (candidate_02 entry point).
```