hakmem/PHASE2_PERF_ANALYSIS.md

HAKMEM Allocator - Phase 2 Performance Analysis

Quick Summary

Metric	Phase 1	Phase 2	Change
Throughput	72M ops/s	79.8M ops/s	+10.8% ✓
Cycles	78.6M	72.2M	-8.1% ✓
Instructions	167M	153M	-8.4% ✓
Branches	36M	23M	-36% ✓
Branch Misses	921K (2.56%)	1.02M (4.43%)	+73% ✗
L3 Cache Misses	173K (9.28%)	216K (10.28%)	+25% ✗
dTLB Misses	N/A	41 (0.01%)	Excellent! ✓

Top 5 Hotspots (Phase 2, 628 samples)

1. malloc() - 36.51% CPU time

   • Function overhead (prologue/epilogue): ~18%
   • Lock operations: 5.05%
   • Initialization checks: ~15%

2. main() - 30.51% CPU time

   • Benchmark loop overhead (not allocator)

3. free() - 19.66% CPU time

   • Lock operations: 3.29%
   • Cached variable checks: ~15%
   • Function overhead: ~10%

4. clear_page_erms (kernel) - 9.31% CPU time

   • Page fault handling

5. irqentry_exit_to_user_mode (kernel) - 5.33% CPU time

   • Kernel exit overhead

Phase 3 Optimization Targets (Ranked by Impact)

🔥 Priority 1: Fast-Path Inlining (Expected: +5-8%)

Target: Reduce malloc/free from 56% → ~33% CPU time

• Inline hot paths to eliminate function call overhead
• Remove stats counters from production builds
• Cache initialization state in TLS

🔥 Priority 2: Branch Optimization (Expected: +3-5%)

Target: Reduce branch misses from 1.02M → <700K

• Apply Profile-Guided Optimization (PGO)
• Add LIKELY/UNLIKELY hints
• Reduce branches in fast path from ~15 to 5-7

🔥 Priority 3: Cache Optimization (Expected: +2-4%)

Target: Reduce L3 misses from 216K → <180K

• Align hot structures to cache lines
• Add prefetching in allocation path
• Compact metadata structures

🎯 Priority 4: Remove Init Overhead (Expected: +2-3%)

• Cache g_initialized/g_enable checks in TLS
• Use constructor attributes more aggressively

🎯 Priority 5: Reduce Lock Contention (Expected: +1-2%)

• Move stats to TLS, aggregate periodically
• Eliminate atomic ops from fast path

🎯 Priority 6: Optimize TLS Operations (Expected: +1-2%)

• Reduce TLS reads/writes from ~10 to ~4 per operation
• Cache TLS values in registers

Expected Phase 3 Results

Target Throughput: 87-95M ops/s (+9-19% improvement)

Metric	Phase 2	Phase 3 Target	Change
Throughput	79.8M ops/s	87-95M ops/s	+9-19%
malloc CPU	36.51%	~22%	-40%
free CPU	19.66%	~11%	-44%
Branch misses	4.43%	<3%	-32%
L3 cache misses	10.28%	<8%	-22%

Key Insights

✅ What Worked in Phase 2

1. SuperSlab size increase (64KB → 512KB): Dramatically reduced branches (-36%)
2. Amortized initialization: memset overhead dropped from 6.41% → 1.77%
3. Virtual memory optimization: TLB miss rate is excellent (0.01%)

❌ What Needs Work

1. Branch prediction: Miss rate doubled despite fewer branches
2. Cache pressure: Larger SuperSlabs increased L3 misses
3. Function overhead: malloc/free dominate CPU time (56%)

🤔 Surprising Findings

1. Cross-calling pattern: malloc/free call each other 8-12% of the time

   • Thread-local cache flushing
   • Deferred release operations
   • May benefit from batching

2. Kernel overhead increased: clear_page_erms went from 2.23% → 9.31%

   • May need page pre-faulting strategy

3. Main loop visible: 30.51% CPU time

   • Benchmark overhead, not allocator
   • Real allocator overhead is ~56% (malloc + free)

Files Generated

• perf_phase2_stats.txt - perf stat -d output
• perf_phase2_symbols.txt - Symbol-level hotspots
• perf_phase2_callgraph.txt - Call graph analysis
• perf_phase2_detailed.txt - Detailed counter breakdown
• perf_malloc_annotate.txt - Assembly annotation for malloc()
• perf_free_annotate.txt - Assembly annotation for free()
• perf_analysis_summary.txt - Detailed comparison with Phase 1
• phase3_recommendations.txt - Complete optimization roadmap

How to Use This Data

For Quick Reference

cat perf_phase2_stats.txt        # See overall metrics
cat perf_phase2_symbols.txt      # See top functions

For Deep Analysis

cat perf_malloc_annotate.txt     # See assembly-level hotspots in malloc
cat perf_free_annotate.txt       # See assembly-level hotspots in free
cat perf_analysis_summary.txt    # See Phase 1 vs Phase 2 comparison

For Planning Phase 3

cat phase3_recommendations.txt   # See ranked optimization opportunities

To Re-run Analysis

# Quick stat
perf stat -d ./bench_random_mixed_hakmem 1000000 256 42

# Detailed profiling
perf record -F 9999 -g ./bench_random_mixed_hakmem 5000000 256 42
perf report --stdio --no-children --sort symbol

Next Steps

1. Week 1: Implement fast-path inlining + remove stats locks (Expected: +8-10%)
2. Week 2: Apply PGO + branch hints (Expected: +3-5%)
3. Week 3: Cache line alignment + prefetching (Expected: +2-4%)
4. Week 4: TLS optimization + polish (Expected: +1-3%)

Total Expected: +14-22% improvement → Target: 91-97M ops/s

---

Generated: 2025-11-28 Phase: 2 → 3 transition Baseline: 72M ops/s → Current: 79.8M ops/s → Target: 87-95M ops/s