Hypothesis — expA (probe reliability / the noise floor)

Probe reliability — noise floor of linear probes across seeds and datasets. Registered 2026-08-12, before any run. The harness (src/modelmap/probes/linear.py + src/modelmap/stats/replication.py) is implemented and unit-tested against planted structure and pure noise; no real-model run has happened yet.

Hypothesis              : On a 0.5B-class model, per-layer linear-probe maps
                          for simple properties (language ID, code-vs-prose,
                          arithmetic-context) are seed-stable (probe-accuracy
                          SD across ≥5 training seeds < 2 points) but
                          dataset-SENSITIVE: switching prompt corpus moves
                          per-layer accuracy by more than the seed SD on at
                          least a third of layers — i.e. the dataset, not the
                          probe seed, is the dominant noise source
                          (Bolukbasi-consistent).
Falsification criterion : Dies if seed variance ≥ dataset variance across the
                          board (then seeds dominate and every later map needs
                          more seeds, not more corpora); the harness itself is
                          invalid if selectivity on a random-init model
                          exceeds 0.05 (fake structure).
Method                  : Qwen3-0.6B (or Pythia-410M fallback) via mlx-lm;
                          capture residual stream at every layer over ≥2
                          disjoint prompt sets per property (versioned,
                          checksummed promptsets); probe_with_control per
                          (layer, property, seed) for 5 seeds; report
                          selectivity, bootstrap CIs, BH-FDR across the layer
                          scan; replication_rate on top-k layer sets.
Baseline / null         : shuffled-label control (built into every probe);
                          random-init model of identical architecture;
                          random-direction projections.
Result                  : (pending)
Interpretation          : (pending — with explicit confidence level)
Next experiment         : (pending — expA output becomes the first map card)

---

Hypothesis — expA run #2 (structure-borne properties, token-balanced)

Registered 2026-08-12 before the run, after run #1 failed its validity gate (twin at ceiling on lexically separable classes). Promptsets v2 are designed so no token distribution separates the classes:

• word_order — grammatical sentence vs seeded scramble of the SAME words (perfectly token-balanced per item);
• agreement — subject–verb agreement correct vs violated, with is/are balanced across both classes (no single token predicts the class);
• arith_valid — correct vs off-by-small-delta equations (digit distributions overlap across classes).

Hypothesis              : (a) VALIDITY: on token-balanced properties the
                          random-init twin falls to chance — max twin
                          selectivity < 0.05 (mean-pooled AND last-token).
                          (b) SIGNAL: the trained model beats the twin on
                          word_order and agreement — real−twin selectivity
                          > 0.10 on at least 5 layers (FDR-significant).
                          (c) arith_valid: exploratory, no directional
                          prediction at 0.6B.
                          (d) rerun of run #1's core question where
                          accuracy is off ceiling/floor: dataset shift >
                          seed SD on ≥ 1/3 of layers.
Falsification criterion : (a) dies → mean-pooled/last-token reps are
                          unusable even on balanced classes; harness redesign
                          (matched-position tokens) before any atlas map.
                          (b) dies → structural properties are not linearly
                          decodable at 0.6B-4bit: a publishable Level-1
                          negative, and probes/v2 is published as such.
Method                  : capture BOTH mean-pooled and last-token reps in
                          one pass; same grid as run #1 (28 layers × 5 seeds
                          × 2 sets × 3 properties, twin on A sets); word-level
                          class token-overlap certificate stored in the
                          promptset manifest; everything else unchanged.
Baseline / null         : shuffled-label control (every probe); random-init
                          twin; v1 promptsets as positive control (harness
                          must still hit ceiling there — checked on lang_id_A).
Result                  : (pending)
Interpretation          : (pending)
Next experiment         : (pending)

Analysis — expA run #1: the harness failed its own validity gate (and that is the result)

Run: results/expA_probe_reliability/20260812T062605Z/results.json (48 s wall). Model: mlx-community/Qwen3-0.6B-4bit (28 layers) + random-init architecture twin. Grid: 3 properties × 2 disjoint promptsets × 28 layers × 5 seeds, shuffled-label control inside every probe, BH-FDR q=0.05, top-5 replication. Hypothesis and validity criterion registered before the run.

Hypothesis              : probe maps are seed-stable but dataset-sensitive
                          (dataset variance > seed variance on ≥1/3 of layers).
Falsification criterion : seeds dominate; AND harness validity gate: twin
                          (random-init) selectivity must stay < 0.05.
Result                  : PRIMARY HYPOTHESIS UNTESTABLE — ceiling effect:
                          task accuracy 1.000 at every layer, every property,
                          both sets; seed SD = 0.0000; dataset shift = 0.0000.
                          VALIDITY GATE FAILED, decisively: the twin reaches
                          accuracy 1.00 at all 28 layers for all properties
                          (twin max selectivity 0.56–0.88 vs the registered
                          0.05 bar); mean real-minus-twin selectivity is
                          within ±0.06 — i.e. ZERO measurable trained-model
                          signal in the map. 28/28 layers "FDR-significant"
                          in the twin too: the statistics are fine, the
                          measurand is wrong.
Interpretation          : Level 1 FOR THE NEGATIVE CLAIM (controlled,
                          replicated across 5 seeds × 2 sets × 3 properties):
                          on lexically separable classes, per-layer linear
                          probes on mean-pooled representations measure the
                          tokenizer + architecture prior, not learned
                          computation. This is the probing-pitfalls
                          literature (notes §4.1/§4.8 — Hewitt & Liang,
                          Bolukbasi, Dead Salmons) reproduced in our own
                          hands on our own harness, caught by a
                          pre-registered gate rather than by a reviewer.
                          Published as the atlas's first entry
                          (atlas/qwen3-0.6b-4bit/probes/v1,
                          negative_result=true) — maps that fail their nulls
                          are first-class citizens here.
Next experiment         : expA run #2 with (a) promptsets v2 that remove
                          lexical separability (shared vocabulary across
                          classes; structure-borne properties), (b) probe
                          maps published only as REAL−TWIN differentials,
                          (c) last-token representations at matched
                          positions alongside mean-pooling, (d) a
                          class-balanced token-overlap statistic reported
                          per promptset as a difficulty certificate.

Notes

• The ceiling itself was foreseeable in hindsight: template-generated classes differ lexically (FR vs EN vocabulary; def/const tokens; digit tokens), and random features over distinct token distributions are linearly separable. v1 promptsets keep their declared role — they are now the positive control corpus (any harness that fails to reach ceiling on them is broken) rather than a measurement corpus.
• The twin null earns permanent-fixture status: it is cheap (one extra capture pass) and it is the only control in the battery that caught this. Shuffled-label controls passed everywhere (selectivity 0.3–0.7) and would have let the illusion through.
• Wall-clock: full grid in 48 s on M5 Max — noise-floor science at this scale is essentially free; the expensive part was thinking, not compute.

---

Analysis — expA run #2: differential maps work; the twin keeps teaching

Run: results/expA_probe_reliability/20260812T063856Z/results.json (88.6 s). v2 promptsets (structure-borne, token-balanced; overlap certificates in the manifest), mean-pooled AND last-token reps, twin null on A sets, v1 positive control. Hypothesis registered before the run.

Hypothesis (a) VALIDITY : twin max selectivity < 0.05 on token-balanced sets.
Result (a)              : FALSIFIED AGAIN — but the gradient is the finding.
                          word_order: twin acc 0.958–0.963 (sel 0.47–0.59);
                          agreement: twin acc 0.683–0.729 (sel 0.15–0.23);
                          arith_valid: twin acc 0.558–0.579 (sel 0.08–0.12).
                          Even with matched word sets, TOKENIZATION statistics
                          differ between classes (mid-sentence capitals,
                          position-dependent subwords in scrambles; is/are
                          bigram contexts) — the twin mines them. The strict
                          0.05 gate is unreachable for surface-correlated
                          properties; the differential doctrine (v1) is the
                          correct instrument, and v2 applies it.
Hypothesis (b) SIGNAL   : real−twin sel > 0.10 on ≥5 FDR-significant layers
                          for word_order AND agreement.
Result (b)              : SPLIT. agreement PASSES decisively — 25/28 layers
                          (mean pooling; 23/28 last-token), max Δsel +0.379,
                          real acc up to 0.967 vs twin 0.729. word_order
                          FAILS — 2/28 layers (0 last-token), max Δsel
                          +0.129: grammatical-vs-scrambled is essentially
                          null-dominated; NOT valid evidence of learned
                          syntax at this granularity.
Hypothesis (c) arith    : exploratory.
Result (c)              : POSITIVE SIGNAL — real acc 0.858 (mean) / 0.904
                          (last-token) vs twin 0.558/0.579; 5–6 signal
                          layers; max Δsel +0.358. Equation-validity is
                          linearly decodable above the architecture prior in
                          a 0.6B-4bit model, strongest in last-token reps
                          (consistent with computation completing at "=").
Hypothesis (d) variance : dataset shift > seed SD on ≥1/3 of layers.
Result (d)              : NOT SUPPORTED — 3–8/28 layers across cells. Off
                          ceiling, seed SD is finally nonzero and top-5-layer
                          replication drops to ~0.54 (vs the trivial 1.00 at
                          ceiling): the FIRST real noise-floor numbers of the
                          project. Seed and dataset variance are comparable
                          in this regime; neither dominates.
Positive control        : v1 lang_id_A at ceiling (1.000) — harness intact.
Interpretation          : Level 1 per property, differential claims only.
                          agreement + arith_valid are the project's first
                          POSITIVE maps (published in atlas probes/v2 with
                          per-property verdicts); word_order is flagged
                          null-dominated in the same entry. Twin gate:
                          retired as a binary gate, kept as a mandatory
                          reported baseline — the differential IS the map.
Next experiment         : (1) promote agreement/arith_valid toward Level 2:
                          second method (LEACE erasure damage) on the same
                          sets; (2) expC causal check on the top agreement
                          layers (ablation); (3) noise-floor deepening: more
                          seeds at fixed set to tighten replication CIs
                          (expD); (4) quantization drift of the agreement
                          differential map (candidate_02 entry point).

expA_probe_reliability

Probe reliability — noise floor of linear probes across seeds and datasets

Status: scaffolded — hypothesis to be registered before any benchmark runs (charter §10).