Research gaps — 24 candidate directions

Phase 3 deliverable (charter §6). Each gap carries the six-field block: (1) existing work · (2) what it does not do · (3) why it matters · (4) why it could work · (5) biggest failure risk · (6) smallest falsifying experiment runnable on a 16–64 GB Mac. Gaps marked ⭐ have direct synergy with the ranked shortlist criteria (Phase 4). Sources: state_of_the_art.md, notes §4.1–4.10.

---

Cluster A — Quantization × internals (the thin shelf)

G01 ⭐ Probe-drift atlas across quantization levels

1. Existing: ~5 shallow papers; effects "inconsistent in direction"; no controls.
2. Missing: probe/selectivity drift FP16→Q8→Q4→Q2 with shuffled-label controls, seeds, multiple datasets, on the same checkpoint.
3. Matters: local models are almost always quantized; if probe maps built at FP16 don't transfer, every published FP16-only map is misleading for real deployments.
4. Could work: MLX quantizes natively; probes are L-cost; drift = probe-accuracy delta + direction rotation, both well-defined.
5. Failure risk: drift may be smaller than the seed/dataset noise floor → "no measurable effect" (itself publishable, Level 1).
6. Falsifier: Qwen3-0.6B FP16 vs Q4: 10 probes × 5 seeds × 2 datasets; if |drift| < noise floor for all probes, the "quantization deforms decodability" hypothesis dies at this scale.

G02 ⭐ Circuit survival under quantization

1. Existing: circuit discovery at FP16 only; one 2026 SAE-drift paper; zero circuit-level studies.
2. Missing: does the induction/IOI/EAP-IG circuit found at FP16 survive at Q4/Q2 (same nodes? same faithfulness)?
3. Matters: directly tests whether interpretability results transfer to the models people actually run; a cheap universality probe.
4. Could work: induction heads are L-cost to detect; EAP-IG runs ≤3B; node-set Jaccard + faithfulness curves are clean metrics.
5. Failure risk: gradient-based EAP through quantized weights is awkward — may need dequantized-weight gradients as approximation (documented caveat).
6. Falsifier: GPT-2-small / Qwen3-0.6B: induction-head set at FP16 vs Q4 (MLX). If Jaccard ≈ 1.0 and faithfulness curves overlap within CIs, "quantization deforms circuits" is falsified at this granularity.

G03 ⭐ Weight-geometry predicts quantization sensitivity

1. Existing: quantization methods use activation stats (imatrix, AWQ) for calibration; WeightWatcher predicts behavioral quality.
2. Missing: per-matrix weight statistics (spectra, alpha, kurtosis, outlier counts) predicting where internal drift (G01/G02) concentrates.
3. Matters: a weight-only pre-screen for "which layers will break when you quantize" — instant value for local-model users; bridges expE and expF.
4. Could work: outlier theory (super weights, LLM.int8) already links weight statistics to quantization fragility at the extremes.
5. Failure risk: drift may be diffuse/global rather than matrix-localized → nothing to predict.
6. Falsifier: rank matrices by weight-stat fragility score; correlate with measured per-layer drift from G01. Spearman ρ < 0.3 across 2 models kills it.

G04 Tuned-lens translation distance as a deformation metric

1. Existing: tuned lens per checkpoint; never across quantization levels.
2. Missing: retrain lenses per Q level; use translator distance (‖A_q − A_fp‖, rotation angle) as a cheap per-layer deformation index.
3. Matters: a continuous, per-layer scalar of internal deformation, far cheaper than SAE retraining.
4. Could work: lenses are affine, small, fast to train; distances are well-defined.
5. Failure risk: lens training variance may swamp the signal (lens seed-sensitivity is undocumented — measure first).
6. Falsifier: train 5-seed lens ensembles at FP16 and Q4 on a 0.5B model; if between-Q distance ≤ within-Q seed distance at every layer, the metric is dead.

G05 Quantized-runtime capture (llama.cpp/GGUF toolchain)

1. Existing: llama.cpp eval-callback dumps tensors; imatrix computes activation stats; nothing wraps either for interpretability.
2. Missing: a capture exporter (eval-callback → mmap store with token bookkeeping) enabling probes/ablation on actually-quantized Metal compute, not simulated quantization.
3. Matters: MLX/PyTorch "Q4" simulates quantized weights in fp arithmetic; the deployed runtime differs (kernels, dequant paths). Only this measures reality.
4. Could work: the callback exists, is C-level, works on Metal; the exporter is engineering, not research risk.
5. Failure risk: per-op dump overhead may be 10–100× — could limit corpora to 10⁴ tokens (still enough for probes).
6. Falsifier: build the exporter; if probe accuracy on llama.cpp-Q4 activations ≠ probe accuracy on MLX-simulated-Q4 beyond CI, simulation-based studies (incl. our own G01) are systematically biased — either way we learn something publishable.

Cluster B — Weight-only cartography as pre-screen

G06 ⭐ Weight-only statistics → activation-level ground truth

1. Existing: WeightWatcher (weights→behavioral quality); super weights (one phenomenon); weight-space learning (global properties, tiny models).
2. Missing: the systematic study — per-matrix weight features (spectra, effective rank, alpha, outliers, norms) predicting probe accuracy, localization concentration, ablation damage on the same checkpoints.
3. Matters: if it works, any model gets a free first map in CPU-minutes, and expensive activation methods get targeted where they matter (the accuracy/cost frontier, Q4).
4. Could work: MLP-KV theory + LASER + super-weight results all say weight structure carries functional signal; nobody has regressed it against activation maps.
5. Failure risk: weight statistics may only predict global quality, not where things live (Friedman illusion: ID-proxy ≠ mechanism).
6. Falsifier: on Pythia-410M: 20 weight features per matrix vs per-layer probe accuracy for 10 properties. If cross-validated R² < 0.2 for all properties, weight-only pre-screens die (a Level-1 negative worth publishing).

G07 Base↔instruct delta atlas

1. Existing: task arithmetic, DARE sparsity, crosscoder diffing (activation-side), Watch-the-Weights (monitoring).
2. Missing: a systematic atlas of instruct deltas: per-matrix delta norms, delta effective rank, delta spectra across ≥3 families, versioned and comparable.
3. Matters: "what does instruction-tuning touch, structurally?" is a question every fine-tuner has; pure weight arithmetic = free at 7–14B.
4. Could work: deltas are known-sparse and low-rank-ish; the descriptive statistics are trivially computable; cross-family comparison is the novelty.
5. Failure risk: descriptive-only risk — without behavioral correlation it's stamp collecting (mitigate: correlate delta concentration with steering/refusal behavior deltas).
6. Falsifier: if delta statistics are indistinguishable across families/matrices (uniform noise), there is no structure to map. Qwen3 base/instruct pair, one afternoon.

G08 Super-weight / outlier cartography across families and quantization

1. Existing: super weights identified on ~10 models (Apple 2024); sinks/massive activations literature.
2. Missing: a versioned catalog: outlier coordinates per model+quantization, causal verification (zero → PPL), do quantization schemes preserve them, do they collide with LoRA/edit targets?
3. Matters: single weight-visible scalars with catastrophic causal load are the strongest known weight→function bridge; atlas-perfect objects (small, verifiable, comparable).
4. Could work: detection is near-free; verification is minutes per model in MLX.
5. Failure risk: low — the phenomenon replicates; risk is shallowness (a catalog without new science). Mitigate by testing preservation under Q2 and interaction with G03.
6. Falsifier: if super-weight coordinates are not stable across quantization of the same checkpoint, the "weight-visible" premise weakens (worth knowing either way).

G09 Cheap maps → pruning/redundancy tolerance

1. Existing: angular-similarity depth pruning (ShortGPT/Gromov); WeightWatcher alphas.
2. Missing: do weight-only features predict which layers are deletable before running the calibration pass; does redundancy predict capability-specific damage (reasoning vs perplexity)?
3. Matters: pruning tolerance is Q5 utility; also feeds localvm (skip-loading redundant blocks).
4. Could work: redundancy is measurable ground truth at L-cost; regression setup identical to G06.
5. Failure risk: redundancy may be purely activation-geometric (invisible in weights).
6. Falsifier: same protocol as G06 with layer-deletion damage as target; R² < 0.2 kills it.

Cluster C — Replication and method-agreement as first-class science

G10 ⭐ Noise-floor atlas (seed/dataset variance of every map type)

1. Existing: isolated numbers — ~30% SAE seed overlap, 1–5% universal neurons, probe-control literature.
2. Missing: a systematic noise-floor table: for each map type (probe, head-importance, feature, circuit), the distribution of map-to-map distance under seed/data resampling, on fixed checkpoints.
3. Matters: it is the denominator of every claim in the atlas; the field cites the SAE number precisely because nothing else exists.
4. Could work: replication is embarrassingly parallel (MacLustr cluster!); metrics (Jaccard, rank correlation) are standard.
5. Failure risk: none scientific — cost only. This is expA/expD formalized.
6. Falsifier: n/a (it is the calibration); the interesting outcome is which map types have floors low enough to build on.

G11 Method-agreement scoring (map = intersection of techniques)

1. Existing: MIB compares methods on benchmarks; pairwise agreement studies are ad hoc.
2. Missing: agreement as the map: publish per-unit agreement across probe/erasure/patching (Level-2 = method-robust made operational and quantitative).
3. Matters: operationalizes our confidence taxonomy; agreement rate itself calibrates how much any single method over-claims.
4. Could work: all three methods are L–M cost; agreement statistics are simple; charter already demands ≥2 techniques for Level 2.
5. Failure risk: agreement near chance everywhere (charter §17 failure criterion) — would force a pivot but is a Level-1 publishable negative.
6. Falsifier: language-identity capability on a 1B bilingual model: probe-top-k vs LEACE-damage vs ablation-damage unit sets. Cohen's κ ≤ 0 across methods = methods measure different things; the "map" concept needs redefinition.

G12 Correlational→causal survival rate as a published number

1. Existing: anecdotes (Hase; IOI ~87%); no systematic conversion rates per technique.
2. Missing: run the pipeline correlational-finding → causal test at scale and publish survival rates per (technique, capability, model).
3. Matters: the single number that tells the field how much to discount correlational maps; calibrates all our Level assignments (expC).
4. Could work: we generate correlational findings in bulk anyway (G10/G11); verification is patching/ablation at M-cost.
5. Failure risk: survival rate may depend heavily on thresholds — pre-register thresholds to avoid garden-of-forking-paths.
6. Falsifier: n/a (measurement); risk-case is rates so threshold-sensitive they're meaningless — detectable by sensitivity analysis.

G13 Ablation-scheme robustness curves as the artifact standard

1. Existing: Miller et al. showed faithfulness isn't robust; recommendations exist; nobody publishes curves.
2. Missing: every importance/faithfulness claim shipped as a curve over {zero, mean, resample} × {noising, denoising} instead of one number.
3. Matters: turns a known critique into a schema requirement; scheme-sensitivity itself becomes a stability metric per map.
4. Could work: auto-circuit already implements efficient multi-scheme ablation; cost multiplier ≈ 6× on L–M experiments.
5. Failure risk: none scientific; adoption/curation cost only.
6. Falsifier: n/a (standard); informative outcome = which map types are scheme-stable.

Cluster D — Cross-model coordinate systems

G14 ⭐ Landmark-anchored cross-model alignment

1. Existing: induction heads universal; universal neurons (seeds); SAE feature matching (noisy); PRH deflated to local alignment.
2. Missing: a coordinate system built from verified landmarks (induction heads, successor heads, super weights, entropy neurons) rather than global similarity metrics; alignment quality scored by landmark correspondence.
3. Matters: Q3 directly; global metrics are debunked-in-part — landmarks are the surviving signal.
4. Could work: landmarks are L-cost to detect and causally verifiable; within-family (shared tokenizer) removes the alignment confound.
5. Failure risk: too few landmarks (~dozens) for a useful coordinate system → coordinates only near landmarks.
6. Falsifier: Qwen3 0.6B vs 1.7B: detect landmarks independently, test whether landmark-anchored layer mapping beats depth-proportional mapping at predicting held-out landmark positions. No improvement = anchoring adds nothing.

G15 Rosetta-mining on LLM pairs

1. Existing: Rosetta Neurons (vision, 8 models); Gurnee universal neurons (same architecture, different seeds).
2. Missing: correlation-mining of shared units across different-size LLMs of one family over a shared corpus.
3. Matters: direct, assumption-light candidate for Q3; complements G14 (dense where G14 is sparse).
4. Could work: method is simple (normalized cross-correlation over shared stimuli); within-family tokenizers align.
5. Failure risk: neuron-level correspondence may be mostly absent above the 1–5% universal set (the honest prior).
6. Falsifier: Pythia-160M vs 410M over 10M shared tokens: if matched-unit rate ≈ shuffled-baseline rate, neuron-level cross-scale correspondence is dead; move to feature-level (SAE) matching.

G16 Cross-scale capability emergence maps (expG formalized)

1. Existing: Tigges (circuits consistent across training/scale, Pythia ≤2.8B); Lieberum (70B case study).
2. Missing: capability-resolved emergence maps: for each capability (code, FR, arithmetic), at which size does localized structure appear, and does its location (relative depth) stay put?
3. Matters: turns "emergence" folklore into mapped, versioned claims; the 70M–2.8B band is fully Mac-feasible.
4. Could work: Pythia/Qwen ladders exist with shared tokenizers; probes + ablation at each size are L–M.
5. Failure risk: capability thresholds may be corpus-artifacts; needs behavioral gating first.
6. Falsifier: if relative-depth location of a capability's probe peak wanders arbitrarily across sizes (no correlation), cross-scale "location" is meaningless and the atlas should never draw it.

Cluster E — Localization science (Q1)

G17 Formal localization-vs-diffusion index

1. Existing: ad hoc top-k ablation curves; "X is localized" claims without a shared measure.
2. Missing: a standard concentration index per (capability, granularity): area-under-damage-curve for top-k units, Gini of importance mass, with nulls and CIs — comparable across models and map types.
3. Matters: Q1's answer must be a number with a distribution, not an adjective; enables cross-capability comparison (code vs FR vs facts).
4. Could work: damage curves are cheap (ablations, L); index properties testable on planted Tracr circuits (known ground truth: fully localized).
5. Failure risk: hydra/backup compensation corrupts single-unit damage curves → must use resample ablation + joint ablations; index may be granularity-dependent (report per granularity).
6. Falsifier: on Tracr models the index must read ≈1 (fully localized); on random-init models ≈0. If it can't separate these, the index is broken before touching real models.

G18 Multilingual (FR/EN) localization atlas

1. Existing: language-agnostic substrate results; multilingual editing transfer studies; no versioned atlas, nothing FR-focused.
2. Missing: enrichment/extraction maps per language on one multilingual model, causally verified, with overlap coefficients (shared vs language-specific units).
3. Matters: clean Q1 instance with personal corpus advantage (native FR promptsets); editing-transfer literature provides validation targets.
4. Could work: Geva pipeline replicates multilingually (2025); 7–8B multilingual models fit a 64 GB Mac.
5. Failure risk: FR/EN too close (shared script/vocabulary) — add a distant-language control (e.g. zh) to calibrate.
6. Falsifier: if FR-specific unit sets under resample ablation damage EN equally (no selectivity), language localization at unit granularity is falsified for this model class.

G19 Editing as perturbation probe (not localization evidence)

1. Existing: ROME→Hase established the negative; editing evaluation critiques.
2. Missing: using edits correctly: edit at layer L, measure the full ripple field (what else moved: probes, features, behavior) as a response map of the network.
3. Matters: turns the field's failed inference into a valid instrument — perturbation-response mapping is standard physics practice.
4. Could work: EasyEdit-class edits are cheap once covariance stats exist; response = pre/post map deltas we already compute.
5. Failure risk: sequential-collapse contamination — use fresh checkpoints per edit; MLX port effort (CUDA-first repos).
6. Falsifier: if response maps are indistinguishable across edit layers (uniform ripple), edits carry no spatial information at all — strengthening Hase into a stronger negative.

Cluster F — Systems and tooling (Q4)

G20 ⭐ Capture cost frontier on Apple Silicon (expH formalized)

1. Existing: zero published macOS numbers (hook overhead MLX vs MPS, mmap vs zarr vs safetensors on APFS, streaming SAE throughput).
2. Missing: the measured "what can you map on which Mac" table; every feasibility estimate in state_of_the_art.md is unverified.
3. Matters: gates everything else; is itself a publishable atlas artifact; extends localvm's SSD results to the capture workload.
4. Could work: benchmarking is deterministic engineering; localvm harness patterns reusable.
5. Failure risk: none scientific; risk is version-fragility (pin MLX/PyTorch versions in manifests).
6. Falsifier: n/a (measurement). Success = published table with CIs; the localvm SSD ceiling (13.1 GB/s ≥256 KiB QD≥4) predicts mmap wins — testable.

G21 Buffer-shuffle vs full-shuffle SAE replication

1. Existing: Anthropic full-shuffles on disk; SAELens/sparsify buffer-shuffle; effect on feature replication unpublished.
2. Missing: does Mac-feasible streaming (small shuffle buffer) degrade seed-replication or feature quality vs full shuffle?
3. Matters: determines whether local SAE training is scientifically viable or systematically biased; informs everyone training SAEs on one GPU.
4. Could work: at 0.5B scale both regimes are runnable locally (full shuffle of a 10⁸-token store fits on SSD).
5. Failure risk: effect may interact with width/architecture — fix one recipe (TopK) first.
6. Falsifier: train 5+5 seeds (buffer vs full shuffle) on Pythia-160M L6; if replication-rate distributions are indistinguishable, buffer streaming is vindicated (positive result for the field).

G22 MLX-native capture/intervention layer

1. Existing: mlxterp (embryonic, no store, no provenance); TransformerLens MPS opt-in; NNsight remote philosophy.
2. Missing: a hardened MLX capture+intervention layer with streaming mmap store, provenance embedding, quantized-model support — the substrate for all our experiments.
3. Matters: without it every experiment pays PyTorch-MPS tax and can't touch quantized models pythonically.
4. Could work: MLX modules are plain Python (wrap-based hooks); laziness makes retention cheap; mlxterp proves the pattern.
5. Failure risk: MLX API churn; mitigation = thin layer + pinned versions + upstream issues.
6. Falsifier: n/a (tooling); acceptance test = capture overhead <2× plain inference at 3B (measured via G20 harness).

Cluster G — Atlas methodology and utility

G23 ⭐ Map cards: machine-readable provenance + confidence schema

1. Existing: datasheets/model cards (data/models); Gemma Scope versioning (no confidence); Neuronpedia hosting (no reproducibility contract).
2. Missing: a schema where every map ships {commit, config, model hash+quant, promptset checksums, hardware manifest, seeds, controls run, replication rate, ablation curves, confidence level} and a regeneration command — enforced by tooling (publish.py refuses incomplete cards).
3. Matters: the charter's core deliverable; §4.10 verified nothing like it exists; it is what makes negative results publishable.
4. Could work: schema design + enforcement is engineering; fields all fall out of Phase 2 conclusions.
5. Failure risk: over-engineering before content exists — co-evolve with the first real maps (expA output = first map card).
6. Falsifier: n/a (standard); external test = a third party regenerates a map from its card alone.

G24 Working-set maps for out-of-core execution (localvm bridge, Q5)

1. Existing: localvm measures SSD/execution side; MoE/routing literature; contextual-sparsity literature (Deja Vu-style) is CUDA-centric and predictor-focused.
2. Missing: maps of token-conditional weight relevance (which blocks matter for which input regime), built with interpretability instruments (ablation, attribution) and validated as prefetch predictions on localvm's engine.
3. Matters: breakthrough criterion E (charter §16): interpretability that materially improves systems decisions — the strongest cross-project payoff.
4. Could work: per-block ablation damage conditioned on input domain is measurable at L–M; localvm provides the consumer and the metric (bytes/token saved at fixed quality).
5. Failure risk: relevance may be too input-specific (no stable per-domain working sets) — itself a decisive result for localvm's architecture choices.
6. Falsifier: on a 3B model: domain-conditioned block-ablation maps (code vs FR prose). If domain-conditional working sets are no smaller than global ones (<10% reduction), per-domain prefetching dies for this model class.

---

Coverage check against charter §6 example categories

quantization-aware maps (G01–G05) · weight-only pre-screens (G06–G09) · cross-model coordinates (G14–G16) · systems-property prediction / localvm bridge (G09, G24) · method-agreement (G11) · longitudinal/cross-scale (G16) · multilingual atlases (G18) · local SAE recipes (G21–G22) · scaled-down causal verification (G12, G17, G19) · uncertainty-first schema (G13, G23). All example categories are covered; G05, G08, G10, G19 fall outside them.

Note on independence

The 24 gaps reduce to roughly five falsification kernels (drift-vs-noise-floor; weight→activation regression; agreement/survival rates; landmark alignment; domain-conditioned ablation) — deliberately: shared kernels mean shared harnesses, and one experiment often feeds several gaps. Phase 4 ranks the gaps, not the kernels.