§4.5 — Weight-space cartography

A1. LASER — layer-selective rank reduction (SVD spectra as intervention targets)

• Key paper: Sharma, Ash, Misra, ICLR 2024, https://arxiv.org/abs/2312.13558 ; code https://github.com/pratyushasharma/laser
• Claims: Replacing individual weight matrices (esp. late-layer MLP down-projections) with low-rank truncations can improve QA accuracy up to 20–30 points.
• Type: Causal on behavior; correlational on interpretation ("noise in high components").
• Cost: SVD of 4096×11008 = seconds on M-series CPU (Accelerate/LAPACK); full per-matrix sweep of a 7B = CPU-minutes; behavioral validation is the expensive part (hours).
• Failure modes: task/matrix-specific gains; (layer, matrix, rank) sweep invites multiple-comparison illusions.
• Epistemic status: Established phenomenon; contested interpretation.

A2. Heavy-tailed spectral analysis / WeightWatcher — closest prior art to the weight-only pre-screen hypothesis

• Key papers: Martin & Mahoney 2019, https://arxiv.org/abs/1901.08278 ; Martin, Peng, Mahoney 2020 (Nature Comms 2021), https://arxiv.org/abs/2002.06716 ; tool: https://github.com/CalculatedContent/WeightWatcher (pip, pure CPU, runs natively on Apple Silicon today)
• Claims: Power-law exponents ("alpha") of per-matrix spectral densities predict relative test quality across model zoos with zero data, zero forward passes; per-layer alpha profiles diagnose over/under-training.
• Cost: cheapest technique surveyed — full 7B scan in minutes on any Mac.
• Failure modes: power-law fitting finicky (fit-range sensitivity); documented Simpson's-paradox failure (https://arxiv.org/abs/2106.00734); predicts trends, not absolute capability; largely untested as a predictor of internal/activation-level structure — that's the modelmap gap.
• Epistemic status: Established for accuracy-trend prediction; contested at fine granularity.

A3. Outlier channels, super weights, massive activations, attention sinks

• Key papers: LLM.int8 outliers: Dettmers et al. 2022, https://arxiv.org/abs/2208.07339 ; massive activations: Sun et al. 2024, https://arxiv.org/abs/2402.17762 ; sinks: Xiao et al. 2023, https://arxiv.org/abs/2309.17453 ; Gu et al. 2024, https://arxiv.org/abs/2410.10781 ; super weights: Yu et al. (Apple) 2024, https://arxiv.org/abs/2411.07191 (https://machinelearning.apple.com/research/the-super-weight)
• Weight-visible vs activation-only: LLM.int8 outliers and massive activations are activation-level; super weights are weight-visible (extreme entries in specific matrices, typically early-layer mlp.down_proj) — a single scalar whose removal collapses perplexity by 3 orders of magnitude and which induces the massive activations. The crucial bridge from weight space to activation space.
• Cost: weight scan CPU-seconds; verification (zero one weight, measure PPL) minutes on a Mac at 7B 4-bit/MLX.
• Epistemic status: Established phenomena; causal story settled for super weights, active for sinks (2026: https://arxiv.org/html/2605.08504 ; survey https://arxiv.org/pdf/2604.10098).

A4. Cross-layer/cross-model similarity (CKA, SVCCA, Procrustes) and CKA's pathologies

• Key papers: CKA: Kornblith et al. 2019, https://arxiv.org/abs/1905.00414 ; SVCCA: https://arxiv.org/abs/1706.05806 ; pathologies: Davari et al., ICLR 2023, https://arxiv.org/abs/2210.16156 ; Ding, Denain, Steinhardt 2021, https://arxiv.org/abs/2108.01661
• Substance: CKA dominated by outlier directions, arbitrarily manipulable without functional change; CKA/CCA disagree and fail sanity checks. Interacts badly with A3: LLM CKA scores can be dominated by sink dimensions unless removed. Canonical CKA needs activations; weight-only analogues (Procrustes/subspace angles between weight matrices, 2026: https://arxiv.org/pdf/2601.10266) exist but are less validated.
• Rule for modelmap: report ≥2 similarity metrics + Ding-style statistical tests; strip outlier dims as a control.
• Epistemic status: Established tool, debunked-in-part as a standalone measure.

A5. Layer redundancy maps (similarity-guided depth pruning)

• Key papers: Gromov et al. 2024, https://arxiv.org/abs/2403.17887 ; ShortGPT: Men et al. 2024, https://arxiv.org/abs/2403.03853
• Claims: Angular similarity between layer input/output identifies deletable deep-layer blocks — an internal depth-redundancy map. Causal (delete-and-measure).
• Note: needs a small calibration set (minutes on Mac). A candidate target that weight-only spectra (A2) might predict — explicit modelmap experiment. Caveat: benchmark loss ≠ capability loss (deep-layer deletion selectively harms reasoning even when perplexity survives).

A6. Model stitching

• Key papers: Lenc & Vedaldi 2015, https://arxiv.org/abs/1411.5908 ; Bansal, Nakkiran, Barak 2021, https://arxiv.org/abs/2106.07682
• Claims: low stitching penalty = functionally interchangeable representations — stronger, task-grounded similarity than CKA. Quasi-causal.
• Cost: train stitching layers — hours per pair on Mac for small LLMs.
• Failure modes: stitcher capacity confounds (expressive stitcher "repairs" dissimilar reps); direction asymmetry. Mostly vision; LLM stitching thin.
• Epistemic status: Established in vision; under-explored for LLMs.

A7. Permutation alignment / Git Re-Basin

• Key papers: Ainsworth et al., ICLR 2023, https://arxiv.org/abs/2209.04836 ; REPAIR: https://arxiv.org/abs/2211.08403 ; Sinkhorn re-basin: https://arxiv.org/abs/2212.12042 ; 2024: https://arxiv.org/abs/2404.06498
• Reality check: transformers align poorly — permutation symmetry broken by residual streams, LayerNorm, head structure; unproven at 7B scale; moot for LLMs anyway (nobody retrains from different seeds). The useful LLM analogue is alignment across sizes/families — open (Q3 / Experiment G).
• Epistemic status: Established for MLPs/ResNets; contested/unproven for LLM transformers.

A8. Mode connectivity

• Key papers: Garipov et al. 2018, https://arxiv.org/abs/1802.10026 ; Frankle et al. 2019, https://arxiv.org/abs/1912.05671 ; Entezari conjecture 2021, https://arxiv.org/abs/2110.06296 ; mechanistic mode connectivity: Lubana et al. 2022, https://arxiv.org/abs/2211.08422
• For modelmap: the cheap artifact is the base↔instruct linear interpolation loss/behavior profile — weight lerp is free, evals are the cost. Loss connectivity ≠ mechanism connectivity.

A9. Task arithmetic / delta analysis

• Key papers: task vectors: Ilharco et al., ICLR 2023, https://arxiv.org/abs/2212.04089 (code https://github.com/mlfoundations/task_vectors — device-agnostic state-dict math) ; TIES: https://arxiv.org/abs/2306.01708 ; DARE: https://arxiv.org/abs/2311.03099 (90%+ of delta entries droppable)
• Claims: θ_ft − θ_base is a manipulable task vector; deltas are highly sparse/redundant. Causal (weight edit → behavior).
• Cost: weight arithmetic free; fully Mac-feasible 7–14B.
• ⚑ Gap: delta maps (base vs instruct: per-matrix delta norms, delta effective rank, delta spectra) have no published systematic atlas — an ideal cheap modelmap artifact.

A10. Model diffing via crosscoders — and 2025 weight-first convergence

• Key papers: Anthropic crosscoders 2024, https://transformer-circuits.pub/2024/crosscoders/index.html ; sparsity artifacts fix: Minder et al. 2025, https://arxiv.org/abs/2504.02922 ; narrow-finetuning traces: https://arxiv.org/abs/2510.13900 ; cross-architecture (2026): https://arxiv.org/pdf/2602.11729 ; post-mortem: https://www.lesswrong.com/posts/xmpauEXEerzYcJKNm/
• Cost: heavy (paired activation capture + SAE-style training). Mac: ≤2B pairs feasible with MLX + streamed stores; 7B pairs at the edge of 64 GB (Experiment H territory).
• Weight-side complements (2025) — direct evidence the field converges on modelmap's weight-first thesis: Watch the Weights https://arxiv.org/pdf/2508.00161 (monitoring fine-tunes from weight diffs alone); Learning to Interpret Weight Differences https://arxiv.org/html/2510.05092v4 ; Steering LMs with Weight Arithmetic https://www.alignmentforum.org/posts/HYTbakdHpxfaCowYp/
• Epistemic status: Promising but contested; methodology moving.

A11. MoE router / expert structure

• Key papers: Mixtral: https://arxiv.org/abs/2401.04088 (routing more syntactic/positional than semantic) ; DeepSeekMoE: https://arxiv.org/abs/2401.06066 ; Myth of Expert Specialization (2026): https://arxiv.org/html/2604.09780v1
• Substance: router weights are weight-only artifacts (expert similarity, router row norms: CPU-minutes); "expert = topic" readings largely debunked; load-balancing losses confound structure. Structure-of-routers is an open, cheap mapping target. Mixtral 8x7B strains 64 GB except 4-bit.

A12. Weight-space learning (weights as data) — and the pre-screen novelty check

• Key sources: Unterthiner et al. 2020, https://arxiv.org/abs/2002.11448 ; Eilertsen et al. 2020, https://arxiv.org/abs/2002.05688 ; Schürholt hyper-representations: https://kschuerholt.github.io/publications/ ; ICLR 2025 Workshop on Weight Space Learning: https://weight-space-learning.github.io/ ; WeightCLIP (2026): https://arxiv.org/html/2607.03551v1
• Claims: model properties predictable from raw weights; but almost all results on tiny CNNs/ViTs — the LLM regime (few, huge models) breaks the model-zoo assumption.

⚑ Novelty check for the core hypothesis (weight-only maps → activation-level properties)

Partially anticipated but not occupied: WeightWatcher predicts behavioral quality (not internal structure); Super Weight links one weight-visible phenomenon to activations; Watch-the-Weights monitors fine-tuning; weight-space learning predicts global properties; LASER flags behaviorally-loaded components; QK/OV composition scores (Elhage et al. 2021 framework) are weight-only circuit hints. Missing and open after aggressive 2025–2026 search: a systematic study of weight-only statistics predicting probe accuracy, localization, quantization drift, or working sets on the same model. This is Experiment E's claim to novelty.