hakmem/docs/analysis/INVESTIGATION_RESULTS.md

# Phase 1 Quick Wins Investigation - Final Results

**Investigation Date:** 2025-11-05
**Investigator:** Claude (Sonnet 4.5)
**Mission:** Determine why REFILL_COUNT optimization failed

---

## Investigation Summary

### Question Asked
Why did increasing `REFILL_COUNT` from 32 to 128 fail to deliver the expected +31% performance improvement?

### Answer Found
**The optimization targeted the wrong bottleneck.**

- **Real bottleneck:** `superslab_refill()` function (28.56% CPU)
- **Assumed bottleneck:** Refill frequency (actually minimal impact)
- **Side effect:** Cache pollution from larger batches (-36% performance)

---

## Key Findings

### 1. Performance Results ❌

| REFILL_COUNT | Throughput | Change | L1d Miss Rate |
|--------------|------------|--------|---------------|
| **32 (baseline)** | **4.19 M ops/s** | **0%** | **12.88%** |
| 64 | 2.69-3.89 M ops/s | -7% to -36% | 14.12% (+10%) |
| 128 | 2.68-4.19 M ops/s | -36% to 0% | 16.08% (+25%) |

**Conclusion:** REFILL_COUNT increases are HARMFUL, not helpful.

---

### 2. Bottleneck Identification 🎯

**Perf profiling revealed:**
```
CPU Time Breakdown:
  28.56% - superslab_refill()        ← THE PROBLEM
   3.10% - [kernel overhead]
   2.96% - [kernel overhead]
   ...    - (remaining distributed)
```

**superslab_refill is 9x more expensive than any other user function.**

---

### 3. Root Cause Analysis 🔍

#### Why REFILL_COUNT=128 Failed:

**Factor 1: superslab_refill is inherently expensive**
- 238 lines of code
- 15+ branches
- 4 nested loops
- 100+ atomic operations (worst case)
- O(n) freelist scan (n=32 slabs) on every call
- **Cost:** 28.56% of total CPU time

**Factor 2: Cache pollution from large batches**
- REFILL=32:  12.88% L1d miss rate
- REFILL=128: 16.08% L1d miss rate (+25% worse!)
- Cause: 128 blocks × 128 bytes = 16KB doesn't fit in L1 (32KB total)

**Factor 3: Refill frequency already low**
- Larson benchmark has FIFO pattern
- High TLS freelist hit rate
- Refills are rare, not frequent
- Reducing frequency has minimal impact

**Factor 4: More instructions, same cycles**
- REFILL=32:  39.6B instructions
- REFILL=128: 61.1B instructions (+54% more work!)
- IPC improves (1.93 → 2.86) but throughput drops
- Paradox: better superscalar execution, but more total work

---

### 4. memset Analysis 📊

**Searched for memset calls:**
```bash
$ grep -rn "memset" core/*.inc
core/hakmem_tiny_init.inc:514:  memset(g_slab_registry, 0, ...)
core/hakmem_tiny_intel.inc:842: memset((void*)g_obs_ready, 0, ...)
```

**Findings:**
- Only 2 memset calls, both in **cold paths** (init code)
- NO memset in allocation hot path
- **Previous perf reports showing memset were from different builds**

**Conclusion:** memset removal would have **ZERO** impact on performance.

---

### 5. Larson Benchmark Characteristics 🧪

**Pattern:**
- 2 seconds runtime
- 4 threads
- 1024 chunks per thread (stable working set)
- Sizes: 8-128B (Tiny classes 0-4)
- FIFO replacement (allocate new, free oldest)

**Implications:**
- After warmup, freelists are well-populated
- High hit rate on TLS freelist
- Refills are infrequent
- **This pattern may NOT represent real-world workloads**

---

## Detailed Bottleneck: superslab_refill()

### Function Location
`/mnt/workdisk/public_share/hakmem/core/hakmem_tiny_free.inc:650-888`

### Complexity Metrics
- Lines: 238
- Branches: 15+
- Loops: 4 nested
- Atomic ops: 32-160 per call
- Function calls: 15+

### Execution Paths

**Path 1: Adopt from Publish/Subscribe** (Lines 686-750)
- Scan up to 32 slabs
- Multiple atomic loads per slab
- Cost: 🔥🔥🔥🔥 HIGH

**Path 2: Reuse Existing Freelist** (Lines 753-792) ← **PRIMARY BOTTLENECK**
- **O(n) linear scan** of all slabs (n=32)
- Runs on EVERY refill
- Multiple atomic ops per slab
- Cost: 🔥🔥🔥🔥🔥 **VERY HIGH**
- **Estimated:** 15-20% of total CPU

**Path 3: Use Virgin Slab** (Lines 794-810)
- Bitmap scan to find free slab
- Initialize metadata
- Cost: 🔥🔥🔥 MEDIUM

**Path 4: Registry Adoption** (Lines 812-843)
- Scan 256 registry entries × 32 slabs
- Thousands of atomic ops (worst case)
- Cost: 🔥🔥🔥🔥🔥 CATASTROPHIC (if hit)

**Path 6: Allocate New SuperSlab** (Lines 851-887)
- **mmap() syscall** (~1000+ cycles)
- Page fault on first access
- Cost: 🔥🔥🔥🔥🔥 CATASTROPHIC

---

## Optimization Recommendations

### 🥇 P0: Freelist Bitmap (Immediate - This Week)

**Problem:** O(n) linear scan of 32 slabs on every refill

**Solution:**
```c
// Add to SuperSlab struct:
uint32_t freelist_bitmap;  // bit i = 1 if slabs[i].freelist != NULL

// In superslab_refill:
uint32_t fl_bits = tls->ss->freelist_bitmap;
if (fl_bits) {
    int idx = __builtin_ctz(fl_bits);  // O(1)! Find first set bit
    // Try to acquire slab[idx]...
}
```

**Expected gain:** +10-15% throughput (4.19 → 4.62-4.82 M ops/s)

---

### 🥈 P1: Reduce Atomic Operations (Next Week)

**Problem:** 32-96 atomic ops per refill

**Solutions:**
1. Batch acquire attempts (reduce from 32 to 1-3 atomics)
2. Relaxed memory ordering where safe
3. Cache scores before atomic acquire

**Expected gain:** +3-5% throughput

---

### 🥉 P2: SuperSlab Pool (Week 3)

**Problem:** mmap() syscall in hot path

**Solution:**
```c
SuperSlab* g_ss_pool[128];  // Pre-allocated pool
// Allocate from pool O(1), refill pool in background
```

**Expected gain:** +2-4% throughput

---

### 🏆 Long-term: Background Refill Thread

**Vision:** Eliminate superslab_refill from allocation path entirely

**Approach:**
- Dedicated thread keeps freelists pre-filled
- Allocation never waits for mmap or scanning
- Zero syscalls in hot path

**Expected gain:** +20-30% throughput (but high complexity)

---

## Total Expected Improvements

### Conservative Estimates

| Phase | Optimization | Gain | Cumulative Throughput |
|-------|--------------|------|----------------------|
| Baseline | - | 0% | 4.19 M ops/s |
| Sprint 1 | Freelist bitmap | +10-15% | 4.62-4.82 M ops/s |
| Sprint 2 | Reduce atomics | +3-5% | 4.76-5.06 M ops/s |
| Sprint 3 | SS pool | +2-4% | 4.85-5.27 M ops/s |
| **Total** | | **+16-26%** | **~5.0 M ops/s** |

### Reality Check

**Current state:**
- HAKMEM Tiny: 4.19 M ops/s
- System malloc: 135.94 M ops/s
- **Gap:** 32x slower

**After optimizations:**
- HAKMEM Tiny: ~5.0 M ops/s (+19%)
- **Gap:** 27x slower (still far behind)

**Conclusion:** These optimizations help, but **fundamental redesign needed** to approach System malloc performance (see Phase 6 goals).

---

## Lessons Learned

### 1. Always Profile First 📊
- Task Teacher's intuition was wrong
- Perf revealed the real bottleneck
- **Rule:** No optimization without perf data

### 2. Cache Effects Matter 🧊
- Larger batches can HURT performance
- L1 cache is precious (32KB)
- Working set + batch must fit

### 3. Benchmarks Can Mislead 🎭
- Larson has special properties (FIFO, stable)
- Real workloads may differ
- **Rule:** Test with diverse benchmarks

### 4. Complexity is the Enemy 🐉
- superslab_refill is 238 lines, 15 branches
- Compare to System tcache: 3-4 instructions
- **Rule:** Simpler is faster

---

## Next Steps

### Immediate Actions (Today)

1. ✅ Document findings (DONE - this report)
2. ❌ DO NOT increase REFILL_COUNT beyond 32
3. ✅ Focus on superslab_refill optimization

### This Week

1. Implement freelist bitmap (P0)
2. Profile superslab_refill with rdtsc instrumentation
3. A/B test freelist bitmap vs baseline
4. Document results

### Next 2 Weeks

1. Reduce atomic operations (P1)
2. Implement SuperSlab pool (P2)
3. Test with diverse benchmarks (not just Larson)

### Long-term (Phase 6)

1. Study System tcache implementation
2. Design ultra-simple fast path (3-4 instructions)
3. Background refill thread
4. Eliminate superslab_refill from hot path

---

## Files Created

1. `PHASE1_REFILL_INVESTIGATION.md` - Full detailed analysis
2. `PHASE1_EXECUTIVE_SUMMARY.md` - Quick reference summary
3. `SUPERSLAB_REFILL_BREAKDOWN.md` - Deep dive into superslab_refill
4. `INVESTIGATION_RESULTS.md` - This file (final summary)

---

## Conclusion

**Why Phase 1 Failed:**

❌ **Optimized the wrong thing** (refill frequency instead of refill cost)
❌ **Assumed without measuring** (refill is cheap, happens often)
❌ **Ignored cache effects** (larger batches pollute L1)
❌ **Trusted one benchmark** (Larson is not representative)

**What We Learned:**

✅ **superslab_refill is THE bottleneck** (28.56% CPU)
✅ **Path 2 freelist scan is the sub-bottleneck** (O(n) scan)
✅ **memset is NOT in hot path** (wasted optimization target)
✅ **Data beats intuition** (perf reveals truth)

**What We'll Do:**

🎯 **Focus on superslab_refill** (10-15% gain available)
🎯 **Implement freelist bitmap** (O(n) → O(1))
🎯 **Profile before optimizing** (always measure first)

**End of Investigation**

---

**For detailed analysis, see:**
- `PHASE1_REFILL_INVESTIGATION.md` (comprehensive report)
- `SUPERSLAB_REFILL_BREAKDOWN.md` (code-level analysis)
- `PHASE1_EXECUTIVE_SUMMARY.md` (quick reference)