Research / research/LOG.md

documentResearch log (append-only)authorSimon-Pierre BouchercreatedTue Aug 11 2026 20:00:00 GMT-0400 (heure avancée de l’Est)statusdraft

modelmap — Research log

Append-only. Every entry records: date/time with timezone, question, experiment, result, interpretation, decision. The entire reasoning process must be auditable.

2026-08-12 00:30 EDT — Project bootstrap

Question. None yet — administrative bootstrap.

Action. Repository initialized from the research charter (CLAUDE.md): full skeleton per charter §3, mandatory author headers on every source file, tools/check_headers.py written first (charter §0.1 rule 3), experiment directories A–H scaffolded with empty seven-field scientific blocks (hypotheses will be registered before any run), and the public platform (site/) built and deployed to https://www.modelmap.io on MacLustr node M3U96b (PM2 + ngrok), mirroring the architecture of the sister project localvm-research (www.localvm.dev) so both research platforms stay coherent.

Decision. Next step is Phase 1: ultra-deep literature research across the ten areas of charter §4 (observational probing, causal methods, feature decomposition, circuits, weight-space cartography, knowledge editing, cross-model comparison, statistics/epistemology, systems, atlas design). No mapping code is written before the state-of-the-art map (Phase 2) exists.

2026-08-12 02:00 EDT — Phase 1 sweep #1: ten-area literature survey

Question. What does the field already know, per charter §4.1–4.10, and which of modelmap's core bets survive first contact with the literature?

Method. Five parallel web-verified survey passes (two charter areas each); every source logged with URL + access date in research/bibliography.md (~200 entries); per-area reading notes written to research/notes/4.X_*.md. This is sweep #1 of Phase 1 — deeper follow-ups (reading full papers behind the highest-stakes claims, backward/forward citation chasing) remain.

Key results (detail in the notes):

  1. The quantization gap is real (Q4/expF bet confirmed). ~5 shallow papers exist on quantization × interpretability; none does probe-drift, circuit survival, Q2/Q3 internals, or GGUF-runtime introspection. Open territory.
  2. The weight-only pre-screen gap is real but partially anticipated (expE). WeightWatcher (behavioral quality from spectra), Super Weight (weight-visible outliers → massive activations), Watch-the-Weights (2025) are adjacent; no systematic weight-only-statistics → activation-level-properties study exists.
  3. Published noise floors to build on: only ~30% of SAE latents replicate across seeds (Paulo & Belrose 2025); only 1–5% of neurons are universal across seeds (Gurnee 2024); global platonic-convergence metrics deflate under calibration (2026). These calibrate expA/expD and our replication-rate metric.
  4. Central methodological caution: localization ≠ editability (Hase 2023); faithfulness scores are not robust to ablation-scheme choice (Miller 2024); subspace patching can hit dormant pathways (Makelov vs Wu, unresolved); random-init baselines + FDR control are mandatory (Dead Salmons 2025).
  5. Apple Silicon tooling is a desert (Q4 confirmed): TransformerLens MPS is opt-in/unstable; editing repos are CUDA-first; only mlxterp (tiny, 2025) targets MLX; llama.cpp eval-callback enables quantized-runtime capture but nothing wraps it; no macOS activation-store or hook-overhead benchmarks exist anywhere.
  6. Atlas prior-art check: no existing artifact host combines versioned entries + regeneration recipes + confidence levels + negative results (verified across Neuronpedia, Neuroscope, Transluce, circuit-tracer, Gemma Scope). "Model atlas" name collision with Horwitz 2025 (inter-model graph, not internals) noted for novelty_check.md.

Interpretation (confidence: this is a literature map, not a result). The charter's bets (quantization maps, weight-only pre-screens, Mac-local tooling, provenance-first atlas) all survive; several have adjacent 2025–2026 work converging on them, so speed matters. The field's own rigor wave (MIB, Tracr/InterpBench validation, statistical-inference framing) supplies ready-made harnesses for our confidence taxonomy.

Decision. Proceed to Phase 2 (research/state_of_the_art.md): organize these techniques into the charter §5 table with epistemic status per technique, then Phase 3 gap generation (≥20 candidates). Validation-first rule adopted: every future pipeline is validated on Tracr/InterpBench-style ground truth before touching a real model.

2026-08-12 02:45 EDT — Phase 2: state-of-the-art map written

Question. Organize Phase 1 into the charter §5 taxonomy: what does each technique actually measure, at what cost on a Mac, with what failure modes and epistemic status — and where are the overlaps and novelty traps?

Action. Wrote research/state_of_the_art.md: ~40 techniques across six families + cross-cutting instruments, each with the eleven-field block (condensed), epistemic flags (established / contested / debunked-in-part), an overlap analysis (6 "secretly the same thing" identities), a list of already-tried combinations, a novelty-trap list, and an estimated Mac cost frontier table (16/32/64 GB) that Experiment H must convert to measured.

Key synthesis judgments (recorded, not yet evidence):

  • Screening/verification split adopted: attribution patching (EAP-IG) screens, real patching verifies; auto-interp labels never exceed Level 1 without intervention scoring.
  • One-estimator-family insight: SAE/transcoder/crosscoder/CLT share the seed-replication pathology — replication findings transfer across them.
  • CKA-class metrics are never standalone evidence: ≥2 metrics + nulls + outlier-stripped variants required.
  • The atlas map-card schema fields fall out of Phase 2: pinned patching protocol, ablation-scheme curves, featurizer class, seed replication rate, per-dataset agreement.

Decision. Proceed to Phase 3 (research/research_gaps.md): ≥20 substantially different candidate directions, each with the six-field gap block and a smallest falsifying experiment runnable on a Mac. Seeds from Phase 2: quantization × internals; weight-only pre-screens vs activation ground truth; replication-rate-first artifacts; quantized-runtime capture; provenance-first atlas schema; cross-model coordinate systems anchored on induction heads.

2026-08-12 03:20 EDT — Phase 3: 24 research gaps registered

Question. Which substantially different directions are (a) genuinely open per the Phase 1/2 evidence, (b) falsifiable with a Mac-scale experiment, (c) worth ranking in Phase 4?

Action. Wrote research/research_gaps.md: 24 gaps (G01–G24) in seven clusters — quantization×internals (G01–G05), weight-only pre-screens (G06–G09), replication/method-agreement science (G10–G13), cross-model coordinates (G14–G16), localization science (G17–G19), systems/tooling (G20–G22), atlas methodology & localvm bridge (G23–G24). Every gap carries the six-field block including its smallest falsifying Mac experiment. All charter §6 example categories are covered; four gaps (G05, G08, G10, G19) fall outside them. Noted that the 24 gaps share ~5 falsification kernels, so harnesses amortize across gaps.

Interpretation. The falsifiers are deliberately harsh (e.g. G06 dies at cross-validated R²<0.2; G11 dies at κ≤0; G17 must separate Tracr from random-init before touching real models). Several gaps are informative in both outcomes (G05's runtime-vs-simulated discrepancy, G24's working-set negative), which is the profile the charter asks for.

Decision. Proceed to Phase 4 (research/candidate_ranking.md): score all 24 gaps on the ten charter axes with written reasoning, select 3–5 prototype candidates. Prior expectation (to be tested by the scoring, not assumed): the quantization cluster and the weight-only pre-screen cluster combine high novelty with low cost; G10/G23 are infrastructure that every other candidate needs regardless of ranking.

2026-08-12 03:50 EDT — Phase 4: candidates ranked and selected

Question. Which of the 24 gaps become prototype candidates, and in what order does everything else get built?

Action. Wrote research/candidate_ranking.md: full 24×10 score table (convention: 10 always favorable, including complexity and risk), grouped written reasoning, and selection. Updated experiments/candidate_01–04 READMEs with assignments.

Selection:

  • candidate_01 — Weight-only pre-screen (G06 + G08 anchor, G03/G09): kill at cross-validated R² < 0.2 on Pythia-410M.
  • candidate_02 — Quantization deformation atlas (G01 + G02, later G04/G05): Qwen3-0.6B FP16-vs-Q4 probe drift vs noise floor.
  • candidate_03 — Localization index (G17 + G11/G12 as measurement layers): must separate Tracr (≈1) from random-init (≈0) first.
  • candidate_04 — localvm working-set bridge (G24 + G09), reduced scope: run the domain-conditioned ablation falsifier before committing.
  • Mandatory substrate first, outside the ranking: G20 (expH cost frontier), G10 (expA/expD noise floors), G23 (map cards v0), G13 (ablation-curves rule, adopted at zero cost).

Interpretation. The infrastructure quartet (G23 87, G20 86, G10 82, G13 75) tops the table by construction — every candidate presupposes it. The three main candidates fail on independent kernels (weights-regression / drift-vs-floor / index-validation), satisfying the charter's demand for genuinely distinct bets; each yields versioned artifacts even when negative. Cross-model coordinate work (G14–G16) is deliberately blocked on noise floors — alignment claims without floors are the field's known failure mode.

Decision. Phases 1–4 complete. Next: Phase 5 (experimental framework) — implement the substrate in order: (1) expH harness (hook overhead MLX vs MPS, storage formats on APFS, streaming throughput) reusing localvm patterns; (2) expA probe-reliability harness with the full controls doctrine; (3) map-cards v0 schema in src/modelmap/atlas enforced by tools/publish.py. Register hypotheses in experiments/micro/expH and expA before their first runs.

2026-08-12 05:30 EDT — Phase 5 substrate + expH run #1 (FIRST MEASURED RESULT)

Question. Build the substrate; then: which activation-store format wins on APFS, and what does capture actually cost on MLX vs PyTorch-MPS?

Built.

  • src/modelmap/atlas/mapcard.py — map-card schema v0 (G23): provenance + per-level requirements (L1: ≥3 seeds, ≥2 promptsets, replication_rate; L2: ≥2 methods; L3: interventions + featurizer class). tools/publish.py is the gate: refuses any atlas entry without a valid card.
  • src/modelmap/stats/replication.py — bootstrap CIs, BH-FDR, selectivity, Jaccard replication_rate (the §8.3 metric).
  • src/modelmap/probes/linear.py — probe_with_control: every probe carries its shuffled-label control by construction. The unit test caught a real bug before any science: rng.shuffle(y[tr]) shuffles a fancy-indexed COPY, so the "control" was training on true labels. Fixed; tests now verify the probe separates planted structure (selectivity>0.3) from pure noise (|selectivity|<0.15). 5/5 tests pass; ruff clean; env = Python 3.14.4, numpy 2.5.2, torch 2.13 (MPS), mlx 0.32, zarr 3.3.

expH run #1 (hypothesis registered BEFORE the run; M5 Max 48 GB, macOS 27.0; results/expH_capture_cost_frontier/20260812T052441Z/):

  • (A) CONFIRMED. Random-batch reads (SAE-shuffle pattern, warm cache): raw-mmap 4.09 GB/s, safetensors(mmap) 4.26, zarr-uncompressed 1.27, zarr-zstd 0.38 → mmap beats zarr 3.2×/10.8×, past the pre-registered 2× bar. safetensors write wins at 8.10 GB/s (single large write).
  • (B) CONFIRMED, asymmetrically. Capture overhead: torch-MPS retain 1.22× plain, +copy+write 1.47×; MLX retain 1.02× (capture nearly free, lazy-graph retention), +copy+write 1.11×. MLX baseline also 1.29× faster than torch-MPS on the identical synthetic 12-layer model.
  • Confidence: Level 0–1 (3 repeats, tight variance, but single machine, warm cache, synthetic model — all declared in advance).
  • Decisions licensed: activation stores = mmap-backed flat files (zarr eliminated for the shuffle pattern pending the cold-cache pass); MLX is the preferred capture backend; capture cost is not the bottleneck — the storage wall is.

Next. expH run #2 (cold cache) and #3 (real 0.5B checkpoint via mlx-lm); then expA first run (hypothesis registered: dataset variance > seed variance; harness validity gate = selectivity<0.05 on random-init). expA's output becomes the first real map card. Replicate expH grid on M3U96a/M2U64 to move the cost table cross-hardware.

2026-08-12 06:30 EDT — expH runs #2 & #3: first FALSIFIED hypothesis + first quantized capture

Questions. Does the warm-cache storage ordering survive a cold cache (run #2, on M3U96a with sudo purge per repetition — also the first cross-hardware point)? Do synthetic capture overheads transfer to a real 4-bit checkpoint (run #3, mlx-lm Qwen3-0.6B-4bit)?

Run #2 — FALSIFIED (the project's first registered-and-killed hypothesis). Cold random-batch reads on M3 Ultra: zarr-uncompressed 0.62 GB/s > safetensors-mmap 0.32 > zarr-zstd 0.19 > raw-mmap 0.14 — the warm ordering (mmap 3.2–10.8× ahead) fully INVERTS. Mechanism: cold mmap is page-fault IO (~8–16 KiB, QD1), exactly the regime localvm's SSD grid measured at ~0.07–0.14 GB/s; zarr read()s 32 MiB chunks — the ≥256 KiB regime where Apple NVMe delivers. Run #1 measured memory, not disk. The rule is IO granularity, not the container. Store design revised: chunk-aligned batched access mandatory for larger-than-RAM stores; run #4 registered (chunk-sorted ≥1 MiB mmap reads vs zarr, cold). Cross-project convergence with localvm expH is strong — the bridge works in both directions.

Run #3 — CONFIRMED, stronger than predicted. Real Qwen3-0.6B-4bit prefill (901 tokens, 28 layers): plain 28.5 ms → retain 28.6 ms (1.004× — capture is free under MLX lazy evaluation) → retain+cast+ mmap-write 36.4 ms (1.28×). First quantized-model activation capture in Python tooling (the §4.9 gap, now demonstrated). Engineering notes: d_model must be inferred from runtime activations (quantized embeddings pack weights); bf16 must be cast in MLX before NumPy.

Decisions. (1) Promote the Tap-wrapper pattern into src/modelmap/capture as the standard MLX capture layer — expA depends on it. (2) The published cost table will carry cache-regime as a first-class dimension (warm ≠ cold columns), and run #2's negative goes on the site's negative-results ledger once the atlas pipeline exists. (3) expA first run is now unblocked (capture layer + probes + stats all in place).

2026-08-12 06:45 EDT — expA run #1: validity gate FAILED → first atlas entry is a negative result

Question. Are per-layer probe maps seed-stable but dataset-sensitive (registered hypothesis), and does the harness pass its registered validity gate (random-init twin selectivity < 0.05)?

Built first. src/modelmap/capture/mlx_capture.py (Tap layer promoted from expH run #3, + random_init_twin architecture null); versioned promptsets v1 (6 files × 240 items, sha256 manifest, template-generated — limitation declared in the manifest itself).

Result (48 s on M5 Max, 2,160 captures + 1,260 probes).

  • Primary hypothesis UNTESTABLE: ceiling everywhere (acc 1.000, seed SD 0.0000, dataset shift 0.0000, all 28 layers, all 3 properties, both sets).
  • Validity gate FAILED, decisively: the random-init twin also reaches acc 1.00 at every layer (twin max selectivity 0.56–0.88 vs the 0.05 bar; mean real−twin selectivity within ±0.06). The probe map contains zero measurable trained-model signal — it reads the tokenizer + architecture prior. The shuffled-label control passed everywhere and would NOT have caught this; only the twin null did.

Interpretation. Level 1 for the negative claim (5 seeds × 2 sets × 3 properties). The probing-pitfalls literature reproduced in our own hands, caught by a pre-registered gate. Exactly what expA exists for: the noise floor turned out to be a ceiling.

Published. First atlas entry: atlas/qwen3-0.6b-4bit/probes/v1 with map.json (all per-layer profiles incl. twin), provenance.json, mapcard.json (negative_result=true, replication 1.00, Level 1) and confidence.md — validated by tools/publish.py and exported to the site. The atlas begins with a negative result, by design.

Decisions. (1) Probe maps are publishable only as REAL−TWIN differentials from now on — schema rule added to the map-card doctrine. (2) Promptsets v1 are re-designated as the positive-control corpus; v2 must remove lexical separability (shared vocabulary, structure-borne properties). (3) The twin null becomes a permanent fixture of every observational map. Next: expA run #2 on v2 promptsets.