---
project: modelmap
document: expH_capture_cost_frontier — analysis (run #1)
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Analysis — expH run #1 (storage formats + hook overhead)

Run: `results/expH_capture_cost_frontier/20260812T052441Z/results.json`
Hardware: Apple M5 Max, 48 GB unified memory, macOS 27.0 · torch 2.13.0 (MPS) ·
MLX 0.32.0 · Python 3.14.4. Hypothesis registered before the run in
`hypothesis.md`; config, seed, commit and full manifest embedded in the JSON.

```text
Hypothesis              : (A) raw mmap ≥ 2× zarr on random-batch reads (warm);
                          (B) retain-capture < 2× plain inference on both
                          backends; copy+write dominates capture cost.
Falsification criterion : (A) dies at ratio < 1.25×; (B) dies at retain > 2×
                          or total > 3×.
Method / Baseline       : as registered (see hypothesis.md) — 3 repeats/cell,
                          plain-forward baselines, sequential-scan references.
Result                  : (A) CONFIRMED. Random-batch reads: raw-mmap 4.09 GB/s,
                          safetensors(mmap) 4.26, zarr-uncompressed 1.27,
                          zarr-zstd 0.38 → mmap/zarr ratio 3.2× (uncompressed)
                          and 10.8× (zstd), both past the 2× bar.
                          Writes: safetensors 8.10 GB/s (single large write),
                          raw-mmap chunked 2.82, zarr-uncompressed 1.49,
                          zarr-zstd 0.54. Seq scans: 3.9–7.3 GB/s all formats.
                          (B) CONFIRMED, asymmetrically. torch-MPS: plain
                          46.7 ms/fwd (±1.6) → retain 57.2 (1.22×) →
                          retain+copy+write 68.8 (1.47×). MLX: plain 36.2
                          (±2.6) → retain 36.9 (1.02×) → +copy+write 40.2
                          (1.11×). Retain never exceeds 1.25×; totals never
                          exceed 1.5× — far under the 2×/3× kill lines.
                          The "copy+write dominates" sub-claim holds on MLX
                          (+3.3 ms vs +0.7 ms retain) but on torch-MPS the
                          two costs are comparable (+10.5 ms retain,
                          +11.6 ms copy+write) — retain itself is expensive
                          on MPS, matching its eager hook materialization.
                          Bonus observation (not registered, Level 0): MLX
                          baseline is 1.29× faster than torch-MPS on the
                          identical architecture, and MLX capture is nearly
                          free (2%), consistent with lazy-graph retention.
Interpretation          : Level 0–1. Within-machine replication only
                          (3 repeats/cell, tight variance ≤ ±4 ms); single
                          hardware, single size class, warm cache, synthetic
                          model — all declared in advance. Design decisions
                          this licenses NOW: (1) activation stores are
                          mmap-backed flat files (raw or safetensors container
                          for self-description) — zarr is eliminated for the
                          SAE-shuffle pattern unless cold-cache reverses the
                          ordering; (2) MLX is the preferred capture backend,
                          with torch-MPS as the compatibility path; (3)
                          capture-cost is NOT the bottleneck at this scale —
                          the 4 TB/500M-token storage wall (notes §4.9)
                          remains the real constraint.
Next experiment         : run #2 = cold-cache pass (purge/F_NOCACHE) to test
                          whether the mmap>zarr ordering survives; run #3 =
                          same capture modes on a real 0.5B checkpoint via
                          mlx-lm vs transformers-MPS; then replicate the whole
                          grid on ≥2 cluster Macs (M3U96a, M2U64) to promote
                          the cost table toward Level 1 cross-hardware.
```

## Notes

- zarr numbers use chunk = one write-block (4096×4096); different chunk
  geometries could narrow the gap — a registered caveat, not tested here.
- The torch-MPS retain overhead (22%) is per-layer-output retention of
  ~100 MB/forward; capture pipelines that subsample layers will scale it down
  roughly linearly.
- All timing cells and raw seconds are in the results JSON with the hardware
  manifest; the figure-ready aggregation is deferred to the atlas pipeline.

---

# Analysis — expH run #2 (cold cache, second hardware) — **HYPOTHESIS FALSIFIED**

Run: `results/expH_capture_cost_frontier/*_run2_M3U96a/results.json`
Hardware: Apple M3 Ultra (Mac Studio, MacLustr M3U96a), 96 GB, macOS.
Store enlarged to 200k×4096 fp16 (~1.6 GB); `sudo purge` before **every**
timed read repetition. Hypothesis registered before the run.

```text
Hypothesis              : warm ordering survives cold — mmap ≥ 1.5× zarr on
                          random-batch reads.
Falsification criterion : dies if any zarr variant beats mmap (ratio <1.25×).
Result                  : FALSIFIED — the ordering INVERTS cold.
                          Random-batch (cold): zarr-uncompressed 0.62 GB/s,
                          safetensors(mmap) 0.32, zarr-zstd 0.19,
                          raw-mmap 0.14 → zarr-uncompressed beats mmap 4.4×.
                          Sequential (cold): zarr-uncompressed 3.47,
                          zarr-zstd 3.18, raw-mmap 0.64, safetensors 0.53.
Interpretation          : Level 0–1, and the mechanism is legible: cold mmap
                          reads are page-fault-driven — effectively ~8–16 KiB
                          random reads at queue depth 1, which localvm's
                          cold-cache SSD grid already showed run at
                          ~0.07–0.14 GB/s. zarr instead read()s whole 32 MiB
                          chunks, exactly the ≥256 KiB / high-QD regime where
                          Apple NVMe delivers. The warm-cache run #1 numbers
                          measured memory, not disk. Cross-project agreement
                          (modelmap run #2 ↔ localvm expH) is strong
                          convergent evidence for the real rule:
                          ***IO granularity decides, not the container.***
                          Design decision REVISED: for stores larger than
                          RAM, either use chunked containers or access mmap
                          in chunk-aligned batched reads; never row-level
                          random access. Caveat: run #1 was M5 Max, run #2
                          M3 Ultra — the warm-vs-cold contrast within run #2's
                          own machine is what carries the claim; hardware
                          confound noted for the cross-run comparison.
Next experiment         : run #4 (registered idea, not yet run): mmap with
                          chunk-sorted, ≥1 MiB batched reads + F_RDAHEAD vs
                          zarr-uncompressed, cold, same machine — decides the
                          final store design. Also replicate run #2 on the
                          M5 Max laptop for the hardware axis.
```

---

# Analysis — expH run #3 (real 4-bit checkpoint via mlx-lm) — CONFIRMED

Run: `results/expH_capture_cost_frontier/20260812T061742Z/results.json`
Model: mlx-community/Qwen3-0.6B-4bit (28 layers, d_model 1024), prefill of a
901-token prompt, M5 Max 48 GB. Hypothesis registered before the run.

```text
Hypothesis              : retain ≤ 1.15× plain prefill; retain+write ≤ 1.5×.
Result                  : CONFIRMED, stronger than predicted.
                          plain 28.5 ms/prefill (31,644 tok/s) →
                          retain 28.6 ms (1.004× — capture is FREE) →
                          retain+copy+write 36.4 ms (1.28×, 24,733 tok/s).
Interpretation          : Level 0–1 (3 repeats, single machine/model). The
                          synthetic run #1 MLX result (1.02×) transfers to a
                          real quantized checkpoint. This run is also the
                          project's first QUANTIZED-model activation capture
                          in Python — the capability the Phase 1 survey found
                          nowhere (notes §4.9). Engineering note: quantized
                          embeddings pack weight shapes, so d_model must be
                          inferred from a runtime activation, and bf16
                          activations must be cast in MLX before NumPy
                          conversion (numpy has no bfloat16).
Next experiment         : same three modes at 1.7B/4B to trace the overhead
                          curve vs model size; then wire this Tap pattern
                          into src/modelmap/capture as the standard MLX
                          capture layer (expA depends on it).
```