52386401b3
Major Features: - Debug counter infrastructure for Refill Stage tracking - Free Pipeline counters (ss_local, ss_remote, tls_sll) - Diagnostic counters for early return analysis - Unified larson.sh benchmark runner with profiles - Phase 6-3 regression analysis documentation Bug Fixes: - Fix SuperSlab disabled by default (HAKMEM_TINY_USE_SUPERSLAB) - Fix profile variable naming consistency - Add .gitignore patterns for large files Performance: - Phase 6-3: 4.79 M ops/s (has OOM risk) - With SuperSlab: 3.13 M ops/s (+19% improvement) This is a clean repository without large log files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
456 lines
26 KiB
Markdown
456 lines
26 KiB
Markdown
# Phase 6.25-6.27: Architecture Evolution
|
||
|
||
**Visual Guide to Mid Pool Optimization**
|
||
|
||
---
|
||
|
||
## 🏗️ Current Architecture (Phase 6.21)
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ ALLOCATION REQUEST │
|
||
│ (2KB - 52KB, site_id) │
|
||
└────────────────────────┬─────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ TLS Fast Path (Lock-Free) │
|
||
├────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
|
||
│ │ TLS Ring (32) │ │ Active Page A │ │ Active Page B ││
|
||
│ │ LIFO cache │ │ Bump-run alloc │ │ Bump-run alloc ││
|
||
│ │ Per-class │ │ Owner-private │ │ Owner-private ││
|
||
│ │ │ │ 64KB page │ │ 64KB page ││
|
||
│ │ [0] [1] ... [31] │ bump → end │ │ bump → end ││
|
||
│ │ ↑ ↑ ↑ │ │ count: 32/32 │ │ count: 16/32 ││
|
||
│ │ │ │ │ │ │ │ │ ││
|
||
│ │ Pop Pop Pop │ │ [=====> │ │ [=========> ││
|
||
│ └─────────────────┘ └─────────────────┘ └─────────────────┘│
|
||
│ │
|
||
│ ⏱️ Latency: ~10-20 cycles (cache hit) │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
│ Miss (ring empty, pages exhausted)
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ Shared Freelist (Mutex-Protected) │
|
||
├────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ 1. 🔒 Lock shard (class_idx, shard_idx) │
|
||
│ pthread_mutex_lock(&freelist_locks[c][s]) │
|
||
│ ⏱️ ~20-40 cycles (uncontended), ~200+ cycles (contended) │
|
||
│ │
|
||
│ 2. Pop from freelist │
|
||
│ ┌──────────────────────────────────────┐ │
|
||
│ │ Freelist[2KB][17] → [B1]->[B2]->[B3] │ │
|
||
│ └──────────────────────────────────────┘ │
|
||
│ Pop B1, freelist = B2 │
|
||
│ │
|
||
│ 3. 🔓 Unlock shard │
|
||
│ pthread_mutex_unlock(&freelist_locks[c][s]) │
|
||
│ │
|
||
│ ⏱️ Latency: ~50-100 cycles (uncontended), ~300+ cycles (4T) │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
│ Freelist empty
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ Refill Path (mmap syscall) │
|
||
├────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ 1. Drain remote stack (if non-empty) │
|
||
│ Remote MPSC queue → Freelist │
|
||
│ │
|
||
│ 2. ⚠️ BOTTLENECK: refill_freelist() - Allocates 1 page │
|
||
│ ┌──────────────────────────────────────┐ │
|
||
│ │ mmap(64KB) ← SYSCALL │ │
|
||
│ │ Split into 32 blocks (2KB class) │ │
|
||
│ │ Link blocks: B1->B2->...->B32 │ │
|
||
│ │ freelist[c][s] = B1 │ │
|
||
│ └──────────────────────────────────────┘ │
|
||
│ ⏱️ 200-300 cycles (mmap) + 100-150 cycles (split) │
|
||
│ = ~400-500 cycles per refill │
|
||
│ = ~12-15 cycles per block (2KB, 32 blocks/page) │
|
||
│ │
|
||
│ 3. ACE bundle factor (adaptive): 1-4 pages │
|
||
│ But still **1 mmap call per page** → no amortization │
|
||
│ │
|
||
│ ⏱️ Latency: ~400-500 cycles × batch_factor │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
|
||
📊 Performance Impact:
|
||
- 1T: ~40% of alloc time in refill path
|
||
- 4T: ~25% in refill + ~25% in lock contention
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Phase 6.25: Refill Batching Architecture
|
||
|
||
**Change**: Allocate **2-4 pages in batch**, distribute to TLS structures
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ NEW: alloc_tls_page_batch() │
|
||
├────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Input: class_idx, batch_size=2, [slot_a, slot_b], ring, bin │
|
||
│ │
|
||
│ for (i = 0; i < batch_size; i++): │
|
||
│ page = mmap(64KB) ← Still multiple mmaps, but batched │
|
||
│ prefault(page) │
|
||
│ │
|
||
│ if (i < num_slots): │
|
||
│ # Fill TLS active page slot (bump-run ready) │
|
||
│ slots[i].page = page │
|
||
│ slots[i].bump = page │
|
||
│ slots[i].end = page + 64KB │
|
||
│ slots[i].count = 32 (for 2KB class) │
|
||
│ │
|
||
│ else: │
|
||
│ # Fill Ring + LIFO from this page │
|
||
│ for (j = 0; j < blocks_per_page; j++): │
|
||
│ block = page + j * block_size │
|
||
│ if (ring.top < 32): │
|
||
│ ring.push(block) # Fill Ring aggressively │
|
||
│ else: │
|
||
│ lifo.push(block) # Overflow to LIFO │
|
||
│ │
|
||
│ ⏱️ 2 mmaps: 2 × 250 cycles = 500 cycles (vs 2 × 400 = 800) │
|
||
│ ⏱️ Amortized: 500 cycles / 64 blocks = 7.8 cycles/block │
|
||
│ (was 400 / 32 = 12.5 cycles/block, 37% reduction!) │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
|
||
After Batch Refill:
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ TLS State (Fully Stocked) │
|
||
├────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ TLS Ring: [B1][B2][B3]...[B32] ← FULL (32/32) │
|
||
│ Active A: [=====> ] (32 blocks ready) │
|
||
│ Active B: [=====> ] (32 blocks ready) │
|
||
│ LIFO: B65->B66->...->B128 (overflow, if batch=4) │
|
||
│ │
|
||
│ Next 96 allocations: TLS fast path (no lock!) │
|
||
│ ⏱️ ~10-20 cycles each │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
|
||
📊 Performance Gain:
|
||
✅ Refill frequency: 1000/sec → 250-500/sec (1T)
|
||
✅ Amortized syscall cost: 12.5 → 7.8 cycles/block (37% reduction)
|
||
✅ More TLS hits: Ring full more often
|
||
✅ Expected: +10-15% (Mid 1T)
|
||
```
|
||
|
||
---
|
||
|
||
## 🔓 Phase 6.26: Lock-Free Freelist Architecture
|
||
|
||
**Change**: Replace mutex with atomic CAS on freelist head
|
||
|
||
### Before (Mutex-Based)
|
||
|
||
```
|
||
Thread 1 (allocating): Thread 2 (allocating):
|
||
pthread_mutex_lock(lock) pthread_mutex_lock(lock)
|
||
↓ Acquired ↓ BLOCKED (waiting...)
|
||
head = freelist[c][s] ↓ Spinning...
|
||
block = head ↓ Spinning... (200+ cycles)
|
||
freelist[c][s] = block->next ↓
|
||
pthread_mutex_unlock(lock) ↓ Acquired (finally!)
|
||
↓ head = freelist[c][s]
|
||
↓ ...
|
||
pthread_mutex_unlock(lock)
|
||
|
||
⏱️ Thread 2 latency: ~200-400 cycles (contended)
|
||
```
|
||
|
||
### After (Lock-Free CAS)
|
||
|
||
```
|
||
Thread 1 (allocating): Thread 2 (allocating):
|
||
old_head = atomic_load(freelist) old_head = atomic_load(freelist)
|
||
block = old_head block = old_head
|
||
CAS(freelist, old_head, block->next) CAS(freelist, old_head, block->next)
|
||
↓ SUCCESS ✅ ↓ FAIL (ABA: T1 changed it)
|
||
↓ Retry:
|
||
↓ old_head = atomic_load(...)
|
||
↓ block = old_head
|
||
↓ CAS(...) ✅ SUCCESS
|
||
|
||
⏱️ Thread 2 latency: ~20-30 cycles (1 retry typical)
|
||
⏱️ No blocking, forward progress guaranteed
|
||
```
|
||
|
||
### Lock-Free Pop Implementation
|
||
|
||
```c
|
||
static inline PoolBlock* freelist_pop_lockfree(int class_idx, int shard_idx) {
|
||
uintptr_t old_head;
|
||
PoolBlock* block;
|
||
|
||
do {
|
||
// Load current head (atomic)
|
||
old_head = atomic_load_explicit(
|
||
&g_pool.freelist_head[class_idx][shard_idx],
|
||
memory_order_acquire);
|
||
|
||
if (!old_head) return NULL; // Empty, fast exit
|
||
|
||
block = (PoolBlock*)old_head;
|
||
|
||
// CAS: Try to swing head to next
|
||
// If another thread modified head, CAS fails → retry
|
||
} while (!atomic_compare_exchange_weak_explicit(
|
||
&g_pool.freelist_head[class_idx][shard_idx],
|
||
&old_head, // Expected value
|
||
(uintptr_t)block->next, // New value
|
||
memory_order_release, // Success ordering
|
||
memory_order_acquire)); // Failure ordering
|
||
|
||
// Update count (relaxed, non-critical)
|
||
atomic_fetch_sub(&g_pool.freelist_count[class_idx][shard_idx], 1,
|
||
memory_order_relaxed);
|
||
|
||
return block;
|
||
}
|
||
```
|
||
|
||
### Data Structure Evolution
|
||
|
||
```
|
||
BEFORE (Phase 6.21):
|
||
┌────────────────────────────────────────┐
|
||
│ g_pool: │
|
||
│ PoolBlock* freelist[7][64] │
|
||
│ PaddedMutex locks[7][64] │ ← 448 mutexes!
|
||
│ atomic nonempty_mask[7] │
|
||
│ atomic remote_head[7][64] │ ← Already lock-free
|
||
│ atomic remote_count[7][64] │
|
||
└────────────────────────────────────────┘
|
||
|
||
AFTER (Phase 6.26):
|
||
┌────────────────────────────────────────┐
|
||
│ g_pool: │
|
||
│ atomic_uintptr_t freelist_head[7][64]│ ← Lock-free!
|
||
│ atomic_uint freelist_count[7][64] │ ← Lock-free counter
|
||
│ atomic nonempty_mask[7] │
|
||
│ atomic remote_head[7][64] │
|
||
│ atomic remote_count[7][64] │
|
||
└────────────────────────────────────────┘
|
||
No mutexes! Pure atomic operations.
|
||
```
|
||
|
||
📊 Performance Gain:
|
||
✅ Eliminate 4T lock contention (40% wait → 0%)
|
||
✅ Reduce per-op latency: 50-300 cycles → 20-30 cycles
|
||
✅ Better scalability: O(threads) contention → O(1) atomic ops
|
||
✅ Expected: +15-20% (Mid 4T)
|
||
|
||
---
|
||
|
||
## 🧠 Phase 6.27: Learner Integration Architecture
|
||
|
||
**Change**: Dynamic CAP/W_MAX tuning via background thread
|
||
|
||
### Learner Control Loop (Already Exists!)
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ Learner Thread (1 Hz polling) │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
Every 1 second:
|
||
│
|
||
├─ Snapshot stats:
|
||
│ mid_hits[7], mid_misses[7], mid_refills[7]
|
||
│ large_hits[5], large_misses[5], large_refills[5]
|
||
│
|
||
├─ Compute hit rate per class:
|
||
│ hit_rate = hits / (hits + misses)
|
||
│
|
||
├─ Compare to target (e.g., 0.65 for Mid):
|
||
│ if (hit_rate < 0.62): # Below target - 3%
|
||
│ cap += 8 # Increase inventory
|
||
│ elif (hit_rate > 0.68): # Above target + 3%
|
||
│ cap -= 8 # Reduce inventory (save memory)
|
||
│
|
||
├─ Respect dwell (stability):
|
||
│ Only update if 3 sec elapsed since last change
|
||
│
|
||
├─ Respect limits:
|
||
│ cap = clamp(cap, min=8, max=512)
|
||
│
|
||
├─ Publish new policy (RCU-like):
|
||
│ FrozenPolicy* new_pol = copy(current_pol)
|
||
│ new_pol.mid_cap[i] = cap
|
||
│ atomic_store(&g_frozen_pol, new_pol)
|
||
│ free(old_pol) # GC
|
||
│
|
||
└─ Allocator reads policy:
|
||
const FrozenPolicy* pol = hkm_policy_get()
|
||
Use pol->mid_cap[i] for refill decisions
|
||
|
||
Example Evolution (60-sec run):
|
||
T=0s: CAP = 64 (init), hit_rate = 0.50 (cold start)
|
||
T=3s: CAP = 72 (low hit → increase)
|
||
T=6s: CAP = 80 (still low)
|
||
T=9s: CAP = 88
|
||
T=12s: CAP = 96
|
||
T=15s: CAP = 104, hit_rate = 0.64 (approaching target)
|
||
T=18s: CAP = 104 (stable, within ±3%)
|
||
T=21s: CAP = 104 (stable)
|
||
...
|
||
T=60s: CAP = 104, hit_rate = 0.66 ✅ Converged!
|
||
```
|
||
|
||
### W_MAX Learning (Optional, Canary Deployment)
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ W_MAX Exploration (UCB1 + Canary) │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
|
||
Candidates: [1.40, 1.50, 1.60, 1.70]
|
||
Algorithm: UCB1 (upper confidence bound)
|
||
Score = mean_throughput + √(log(total_pulls) / pulls)
|
||
|
||
Phase 1: Exploration (first 40 sec)
|
||
T=0s: Try 1.40 (never tried) → score = 5.2 M/s
|
||
T=10s: Try 1.50 (never tried) → score = 5.4 M/s
|
||
T=20s: Try 1.60 (never tried) → score = 5.3 M/s
|
||
T=30s: Try 1.70 (never tried) → score = 5.1 M/s (worse, more waste)
|
||
T=40s: Best so far: 1.50 (5.4 M/s)
|
||
|
||
Phase 2: Exploitation (UCB1 selects best)
|
||
T=50s: UCB1 → 1.50 (highest score + bonus)
|
||
T=60s: UCB1 → 1.50 (confidence increasing)
|
||
|
||
Phase 3: Canary Deployment (safe trial)
|
||
T=70s: Canary start: Try 1.60 (second best)
|
||
Measure for 5 sec, compare to baseline (1.50)
|
||
T=75s: Canary result: 5.35 M/s < 5.4 M/s (baseline)
|
||
Revert to 1.50 ✅ (safe rollback)
|
||
|
||
Final: W_MAX = 1.50 (converged to optimal)
|
||
```
|
||
|
||
📊 Performance Gain:
|
||
✅ CAP auto-tuning: Adapt to workload (memory vs speed)
|
||
✅ W_MAX optimization: Find sweet spot (hit rate vs waste)
|
||
✅ DYN1 assignment: Fill gaps dynamically
|
||
✅ Expected: +5-10% (adaptive edge over static config)
|
||
|
||
---
|
||
|
||
## 📊 Cumulative Architecture Impact
|
||
|
||
### Allocation Latency Breakdown (Mid 2KB, 1T)
|
||
|
||
```
|
||
BEFORE (Phase 6.21):
|
||
┌────────────────────────────────────────────────┐
|
||
│ TLS Ring hit (60%): ~15 cycles │ Fast path
|
||
├────────────────────────────────────────────────┤
|
||
│ Active Page hit (20%): ~30 cycles │ Still fast
|
||
├────────────────────────────────────────────────┤
|
||
│ Freelist pop (15%): ~80 cycles │ Lock overhead
|
||
├────────────────────────────────────────────────┤
|
||
│ Refill (5%): ~400 cycles │ BOTTLENECK
|
||
└────────────────────────────────────────────────┘
|
||
Weighted avg: 0.6×15 + 0.2×30 + 0.15×80 + 0.05×400
|
||
= 9 + 6 + 12 + 20 = 47 cycles/alloc
|
||
|
||
AFTER (Phase 6.25 + 6.26 + 6.27):
|
||
┌────────────────────────────────────────────────┐
|
||
│ TLS Ring hit (75%): ~15 cycles │ ↑ Hit rate (learner)
|
||
├────────────────────────────────────────────────┤
|
||
│ Active Page hit (15%): ~30 cycles │ More pages (batch)
|
||
├────────────────────────────────────────────────┤
|
||
│ Freelist pop (8%): ~25 cycles │ Lock-free (6.26)
|
||
├────────────────────────────────────────────────┤
|
||
│ Refill (2%): ~250 cycles │ Batched (6.25)
|
||
└────────────────────────────────────────────────┘
|
||
Weighted avg: 0.75×15 + 0.15×30 + 0.08×25 + 0.02×250
|
||
= 11.25 + 4.5 + 2 + 5 = 22.75 cycles/alloc
|
||
|
||
🎯 Latency reduction: 47 → 23 cycles (52% faster!) ✅
|
||
```
|
||
|
||
### Scalability (4T Contention)
|
||
|
||
```
|
||
BEFORE (Phase 6.21):
|
||
Thread 1: |████████░░| 80% useful work, 20% lock wait
|
||
Thread 2: |██████░░░░| 60% useful work, 40% lock wait
|
||
Thread 3: |██████░░░░| 60% useful work, 40% lock wait
|
||
Thread 4: |██████░░░░| 60% useful work, 40% lock wait
|
||
────────────────────────────────────────────────
|
||
Aggregate: 65% efficiency (35% wasted on contention)
|
||
|
||
AFTER (Phase 6.26, lock-free):
|
||
Thread 1: |██████████| 95% useful work, 5% CAS retry
|
||
Thread 2: |█████████░| 92% useful work, 8% CAS retry
|
||
Thread 3: |█████████░| 92% useful work, 8% CAS retry
|
||
Thread 4: |█████████░| 92% useful work, 8% CAS retry
|
||
────────────────────────────────────────────────
|
||
Aggregate: 92% efficiency (8% CAS overhead)
|
||
|
||
🎯 Efficiency gain: 65% → 92% (42% improvement!) ✅
|
||
```
|
||
|
||
---
|
||
|
||
## 🏁 Final Architecture (Phase 6.27 Complete)
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ HAKMEM Mid Pool (Optimized) │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
|
||
Layer 1: TLS Fast Path (75% hit rate with learner)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Ring Buffer (32 slots) ← Refilled aggressively by batch │
|
||
│ Active Page A (64KB) ← From batch allocator │
|
||
│ Active Page B (64KB) ← From batch allocator │
|
||
└──────────────────────────────────────────────────────────┘
|
||
⏱️ ~15-30 cycles (lock-free, cache-hot)
|
||
|
||
Layer 2: Lock-Free Freelist (20% hit rate)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Atomic freelist_head[7][64] ← CAS-based pop/push │
|
||
│ Atomic freelist_count[7][64] ← Non-blocking counter │
|
||
│ Atomic remote_head[7][64] ← Cross-thread frees │
|
||
└──────────────────────────────────────────────────────────┘
|
||
⏱️ ~20-30 cycles (no mutex, 1-2 CAS retries typical)
|
||
|
||
Layer 3: Batch Refill (5% hit rate)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ alloc_tls_page_batch(class, batch=2) │
|
||
│ → Allocate 2 pages (2 mmaps, batched) │
|
||
│ → Fill Active A, Active B │
|
||
│ → Overflow to Ring (fill to 32/32) │
|
||
│ → Overflow to LIFO (if batch > 2) │
|
||
└──────────────────────────────────────────────────────────┘
|
||
⏱️ ~250 cycles (amortized over 64 blocks = 3.9 cycles/block)
|
||
|
||
Layer 4: Learner (background optimization)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ 1 Hz polling thread │
|
||
│ → Monitor hit rates per class │
|
||
│ → Adjust CAP dynamically (±8 pages) │
|
||
│ → Explore W_MAX (UCB1 + Canary) │
|
||
│ → Publish FrozenPolicy (atomic swap) │
|
||
└──────────────────────────────────────────────────────────┘
|
||
⏱️ Zero hot-path overhead (background thread)
|
||
|
||
────────────────────────────────────────────────────────────────
|
||
Performance vs mimalloc:
|
||
Mid 1T: 4.0 → 5.0 M/s (28% → 34%) ← Still gap (need header opt)
|
||
Mid 4T: 13.8 → 18.5 M/s (47% → 63%) ✅ TARGET ACHIEVED!
|
||
────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
**Last Updated**: 2025-10-24
|
||
**Status**: Architecture Design Complete
|