hakmem/PHASE8_TECHNICAL_ANALYSIS.md

Phase 8 - Technical Analysis and Root Cause Investigation

Executive Summary

Phase 8 comprehensive benchmarking reveals critical performance issues with HAKMEM:

• Working Set 256 (Hot Cache): 9.4% slower than System malloc, 45.2% slower than mimalloc
• Working Set 8192 (Realistic): 246% slower than System malloc, 485% slower than mimalloc

The most alarming finding: HAKMEM experiences 4.8x performance degradation when moving from hot cache to realistic workloads, compared to only 1.5x for System malloc and 1.2x for mimalloc.

Benchmark Results Summary

Working Set 256 (Hot Cache)

Allocator	Avg (M ops/s)	StdDev	vs HAKMEM
HAKMEM Phase 8	79.2	±2.4%	1.00x
System malloc	86.7	±1.0%	1.09x
mimalloc	114.9	±1.2%	1.45x

Working Set 8192 (Realistic Workload)

Allocator	Avg (M ops/s)	StdDev	vs HAKMEM
HAKMEM Phase 8	16.5	±2.5%	1.00x
System malloc	57.1	±1.3%	3.46x
mimalloc	96.5	±0.9%	5.85x

Scalability Analysis

Performance degradation from WS256 → WS8192:

• HAKMEM: 4.80x slowdown (79.2 → 16.5 M ops/s)
• System: 1.52x slowdown (86.7 → 57.1 M ops/s)
• mimalloc: 1.19x slowdown (114.9 → 96.5 M ops/s)

HAKMEM degrades 3.16x MORE than System malloc and 4.03x MORE than mimalloc.

Root Cause Analysis

Evidence from Debug Logs

The benchmark output shows critical issues:

[SS_BACKEND] shared_fail→legacy cls=7
[SS_BACKEND] shared_fail→legacy cls=7
[SS_BACKEND] shared_fail→legacy cls=7
[SS_BACKEND] shared_fail→legacy cls=7

Analysis: Repeated "shared_fail→legacy" messages indicate SuperSlab exhaustion, forcing fallback to legacy allocator path. This happens 4 times during WS8192 benchmark, suggesting severe SuperSlab fragmentation or capacity issues.

Issue 1: SuperSlab Architecture Doesn't Scale

Symptoms:

• Performance collapses from 79.2 to 16.5 M ops/s (4.8x degradation)
• Shared SuperSlabs fail repeatedly
• TLS_SLL_HDR_RESET events occur (slab header corruption?)

Root Causes (Hypotheses):

1. SuperSlab Capacity: Current 512KB SuperSlabs may be too small for WS8192

   • 8192 objects × (16-1024 bytes average) = ~4-8MB working set
   • Multiple SuperSlabs needed → increased lookup overhead

2. Fragmentation: SuperSlabs become fragmented with larger working sets

   • Free slots scattered across multiple SuperSlabs
   • Linear search through slab list becomes expensive

3. TLB Pressure: More SuperSlabs = more page table entries

   • System malloc uses fewer, larger arenas
   • HAKMEM's 512KB slabs create more TLB misses

4. Cache Pollution: Slab metadata pollutes L1/L2 cache

   • Each SuperSlab has metadata overhead
   • More slabs = more metadata = less cache for actual data

Issue 2: TLS Drain Overhead

Debug logs show:

[TLS_SLL_DRAIN] Drain ENABLED (default)
[TLS_SLL_DRAIN] Interval=2048 (default)

Analysis: Even in hot cache (WS256), HAKMEM is 9.4% slower than System malloc. This suggests fast-path overhead from TLS drain checks happening every 2048 operations.

Evidence:

• WS256 should fit entirely in cache, yet HAKMEM still lags
• System malloc has simpler fast path (no drain logic)
• 9.4% overhead = ~7-8 extra cycles per allocation

Issue 3: TLS_SLL_HDR_RESET Events

[TLS_SLL_HDR_RESET] cls=6 base=0x790999b35a0e got=0x00 expect=0xa6 count=0

Analysis: Header reset events suggest slab list corruption or validation failures. This shouldn't happen in normal operation and indicates potential race conditions or memory corruption.

Performance Breakdown

Where HAKMEM Loses Performance (WS8192)

Estimated cycle budget (assuming 3.5 GHz CPU):

• HAKMEM: 16.5 M ops/s = ~212 cycles/operation
• System: 57.1 M ops/s = ~61 cycles/operation
• mimalloc: 96.5 M ops/s = ~36 cycles/operation

Gap Analysis:

• HAKMEM uses 151 extra cycles vs System malloc
• HAKMEM uses 176 extra cycles vs mimalloc

Where do these cycles go?

1. SuperSlab Lookup (~50-80 cycles)

   • Linear search through slab list
   • Cache misses on slab metadata
   • TLB misses on slab pages

2. TLS Drain Logic (~10-15 cycles)

   • Drain counter checks every allocation
   • Branch mispredictions

3. Fragmentation Overhead (~30-50 cycles)

   • Walking free lists
   • Finding suitable free blocks

4. Legacy Fallback (~50-100 cycles when triggered)

   • System malloc/mmap calls
   • Context switches

Competitive Analysis

Why System malloc Wins (3.46x faster)

1. Arena-based design: Fewer, larger memory regions
2. Thread caching: Similar to HAKMEM TLS but better tuned
3. Mature optimization: Decades of tuning
4. Simple fast path: No drain logic, no SuperSlab lookup

Why mimalloc Dominates (5.85x faster)

1. Segment-based design: Optimal for multi-threaded workloads
2. Free list sharding: Reduces contention
3. Aggressive inlining: Fast path is 15-20 instructions
4. No locks in fast path: Lock-free for thread-local allocations
5. Delayed freeing: Like HAKMEM drain but more efficient
6. Minimal metadata: Less cache pollution

Critical Gaps to Address

Gap 1: Fast Path Performance (9.4% slower at WS256)

Target: Match System malloc at hot cache workload Required improvement: +9.4% = +7.5 M ops/s

Action items:

• Profile TLS drain overhead
• Inline critical functions more aggressively
• Reduce branch mispredictions
• Consider removing drain logic or making it lazy

Gap 2: Scalability (246% slower at WS8192)

Target: Get within 20% of System malloc at realistic workload Required improvement: +246% = +40.6 M ops/s (2.46x speedup needed!)

Action items:

• Fix SuperSlab scaling
• Reduce fragmentation
• Optimize SuperSlab lookup (hash table instead of linear search?)
• Reduce TLB pressure (larger SuperSlabs or better placement)
• Profile cache misses

Recommendations for Phase 9+

Phase 9: CRITICAL - SuperSlab Investigation

Goal: Understand why SuperSlab performance collapses at WS8192

Tasks:

1. Add detailed profiling:

   • SuperSlab lookup latency distribution
   • Cache miss rates (L1, L2, L3)
   • TLB miss rates
   • Fragmentation metrics

2. Measure SuperSlab statistics:

   • Number of active SuperSlabs at WS256 vs WS8192
   • Average slab list length
   • Hit rate for first-slab lookup

3. Experiment with SuperSlab sizes:

   • Try 1MB, 2MB, 4MB SuperSlabs
   • Measure impact on performance

4. Analyze "shared_fail→legacy" events:

   • Why do shared slabs fail?
   • How often does it happen?
   • Can we pre-allocate more capacity?

Phase 10: Fast Path Optimization

Goal: Close 9.4% gap at WS256

Tasks:

1. Profile TLS drain overhead
2. Experiment with drain intervals (4096, 8192, disable)
3. Inline more aggressively
4. Add __builtin_expect hints for common paths
5. Reduce branch mispredictions

Phase 11: Architecture Re-evaluation

Goal: Decide if SuperSlab model is viable

Decision point: If Phase 9 can't get within 50% of System malloc at WS8192, consider:

1. Hybrid approach: TLS fast path + different backend (jemalloc-style arenas?)
2. Abandon SuperSlab: Switch to segment-based design like mimalloc
3. Radical simplification: Focus on specific use case (small allocations only?)

Success Criteria for Phase 9

Minimum acceptable improvements:

• WS256: 79.2 → 85+ M ops/s (+7% improvement, match System malloc)
• WS8192: 16.5 → 35+ M ops/s (+112% improvement, get to 50% of System malloc)

Stretch goals:

• WS256: 90+ M ops/s (close to System malloc)
• WS8192: 45+ M ops/s (80% of System malloc)

Raw Data

All benchmark runs completed successfully with good statistical stability (StdDev < 2.5%).

Working Set 256

HAKMEM:   [78.5, 78.1, 77.0, 81.1, 81.2] M ops/s
System:   [87.3, 86.5, 87.5, 85.3, 86.6] M ops/s
mimalloc: [115.8, 115.2, 116.2, 112.5, 115.0] M ops/s

Working Set 8192

HAKMEM:   [16.5, 15.8, 16.9, 16.7, 16.6] M ops/s
System:   [56.1, 57.8, 57.0, 57.7, 56.7] M ops/s
mimalloc: [96.8, 96.1, 95.5, 97.7, 96.3] M ops/s

Conclusion

Phase 8 benchmarking reveals fundamental issues with HAKMEM's current architecture:

1. SuperSlab scaling is broken - 4.8x performance degradation is unacceptable
2. Fast path has overhead - Even hot cache shows 9.4% gap
3. Competition is fierce - mimalloc is 5.85x faster at realistic workloads

Next priority: Phase 9 MUST focus on understanding and fixing SuperSlab scalability. Without addressing this core issue, HAKMEM cannot compete with production allocators.

The benchmark data is statistically robust (low variance) and reproducible. The performance gaps are real and significant.