About this project
modelmap is an independent research project by Simon-Pierre Boucher (contact@spboucher.ai) building a systematic methodology — and the tooling behind it — to discover, measure, and map the internal structure of pretrained open-weight LLMs running locally on consumer Apple Silicon hardware, and publishing the resulting maps as a rigorous, reproducible public atlas on this site.
The ideal outcome is not a gallery of suggestive visualizations. It is a reproducible cartographic standard for open-weight models — every map versioned, provenanced, confidence-labeled, and regenerable by anyone with a Mac — turning “we think the model does X” into “here is the map, its evidence level, and the script that rebuilds it.”
Core research questions
| Q1 — Localization | Where inside a pretrained LLM do capabilities, knowledge domains, languages, and behaviors reside? Localized (layers, heads, neurons, weight blocks, directions) or diffuse? |
|---|---|
| Q2 — Structure | Which stable internal structures exist across inputs — circuits, feature directions, attention patterns, activation manifolds, weight-space geometry — and at what granularity are they real rather than artifacts of the probing method? |
| Q3 — Comparability | Can internal maps be compared across model sizes, checkpoints, quantization levels, and families? Is there a common coordinate system for model internals? |
| Q4 — Cost | Which mapping techniques are feasible on a 16–64 GB Mac, at which model sizes, and what is the accuracy/cost frontier of local interpretability? |
| Q5 — Utility | Do the maps predict anything useful — quantization sensitivity, pruning tolerance, working-set behavior, failure modes, editing targets? |
The evidence standard
- L0
anecdotalsingle run, no controls — never published alone
- L1
correlationalcontrolled, replicated ≥3 seeds, ≥2 datasets
- L2
method-robustLevel 1 + agreement across ≥2 independent techniques
- L3
causalLevel 2 + intervention confirms the claim
Every mapping technique runs with controls: shuffled labels for probes, random-direction baselines for steering, randomly-initialized-model baselines where meaningful, resample vs zero ablations. Structure is never claimed without showing the null. Replication across ≥3 seeds and ≥2 prompt sets, bootstrap confidence intervals, effect sizes, and multiple-comparison correction are mandatory for every published number.
Target hardware
Apple Silicon Mac (M1–M4 family), 16–64 GB unified memory, internal NVMe SSD, Metal GPU sharing memory with the CPU. Local interpretability on consumer hardware is itself an under-served niche: most tooling assumes CUDA clusters. Making rigorous mapping feasible on a Mac is part of the contribution. The stretch target is a full, causally-verified, versioned atlas of a 7B–14B model produced end-to-end on a single 32–64 GB Mac, comparable across ≥2 quantization levels and ≥2 model sizes.
Sister project
localvm-research investigates out-of-core LLM execution on the same hardware class — running models larger than memory. The projects cross-pollinate: modelmap's working-set and localization maps (Q5) feed localvm's execution decisions, and localvm's cost measurements calibrate modelmap's capture-feasibility frontier (Experiment H).
Failure is an acceptable outcome — the charter defines explicit failure criteria, and the log records why an approach died, not just what survived.