hakmem/PHASE6A_BENCHMARK_RESULTS.md

Phase 6-A Benchmark Results

Date: 2025-11-29 Change: Disable SuperSlab lookup debug validation in RELEASE builds File: core/tiny_region_id.h:199-239 Guard: #if !HAKMEM_BUILD_RELEASE around hak_super_lookup() call Reason: perf profiling showed 15.84% CPU cost on allocation hot path (debug-only validation)

---

Executive Summary

Phase 6-A implementation successfully removes debug validation overhead in release builds, but the measured performance impact is significantly smaller than predicted:

• Expected: +12-15% (random_mixed), +8-10% (mid_mt_gap)
• Actual (best 3 of 5): +1.67% (random_mixed), +1.33% (mid_mt_gap)
• Actual (excluding warmup): +4.07% (random_mixed), +1.97% (mid_mt_gap)

Recommendation: HOLD on commit. Investigate discrepancy between perf analysis (15.84% CPU) and benchmark results (~1-4% improvement).

---

Benchmark Configuration

Build Configurations

Baseline (Before Phase 6-A)

make clean
make EXTRA_CFLAGS="-g -O3" bench_random_mixed_hakmem bench_mid_mt_gap_hakmem
# Note: Makefile sets -DHAKMEM_BUILD_RELEASE=1 by default
# Result: SuperSlab lookup ALWAYS enabled (no guard in code yet)

Phase 6-A (After)

git stash pop  # Restore Phase 6-A changes
make clean
make EXTRA_CFLAGS="-g -O3" bench_random_mixed_hakmem bench_mid_mt_gap_hakmem
# Note: Makefile sets -DHAKMEM_BUILD_RELEASE=1 by default
# Result: SuperSlab lookup DISABLED (guarded by #if !HAKMEM_BUILD_RELEASE)

Benchmark Parameters

• Iterations: 1,000,000 operations per run
• Working Set: 256 blocks
• Seed: 42 (reproducible)
• Runs: 5 per configuration
• Suppression: 2>/dev/null to exclude debug output noise

---

Raw Results

bench_random_mixed (Tiny workload, 16B-1KB)

Baseline (Before Phase 6-A, SuperSlab lookup ALWAYS enabled)

Run 1: 53.81 M ops/s
Run 2: 53.25 M ops/s
Run 3: 53.56 M ops/s
Run 4: 49.41 M ops/s
Run 5: 51.41 M ops/s
Average: 52.29 M ops/s
Stdev: 1.86 M ops/s

Phase 6-A (Release build, SuperSlab lookup DISABLED)

Run 1: 39.11 M ops/s  ⚠️ OUTLIER (warmup)
Run 2: 53.30 M ops/s
Run 3: 56.28 M ops/s
Run 4: 52.79 M ops/s
Run 5: 53.72 M ops/s
Average: 51.04 M ops/s (all runs)
Stdev: 6.80 M ops/s (high due to outlier)
Average (excl. Run 1): 54.02 M ops/s

Outlier Analysis: Run 1 is 27.6% slower than the average of runs 2-5, indicating a warmup/cache-cold issue.

---

bench_mid_mt_gap (Mid MT workload, 1KB-8KB)

Baseline (Before Phase 6-A, SuperSlab lookup ALWAYS enabled)

Run 1: 41.70 M ops/s
Run 2: 37.39 M ops/s
Run 3: 40.91 M ops/s
Run 4: 40.53 M ops/s
Run 5: 40.56 M ops/s
Average: 40.22 M ops/s
Stdev: 1.65 M ops/s

Phase 6-A (Release build, SuperSlab lookup DISABLED)

Run 1: 41.49 M ops/s
Run 2: 41.81 M ops/s
Run 3: 41.51 M ops/s
Run 4: 38.43 M ops/s
Run 5: 40.78 M ops/s
Average: 40.80 M ops/s
Stdev: 1.38 M ops/s

Variance Analysis: Both baseline and Phase 6-A show similar variance (~3-4 M ops/s spread), suggesting measurement noise is inherent to this benchmark.

---

Statistical Analysis

Comparison 1: All Runs (Conservative)

Benchmark	Baseline	Phase 6-A	Absolute	Relative	Expected	Result
random_mixed	52.29 M	51.04 M	-1.25 M	-2.39%	+12-15%	❌ FAIL
mid_mt_gap	40.22 M	40.80 M	+0.59 M	+1.46%	+8-10%	❌ FAIL

Comparison 2: Excluding First Run (Warmup Correction)

Benchmark	Baseline	Phase 6-A	Absolute	Relative	Expected	Result
random_mixed	51.91 M	54.02 M	+2.11 M	+4.07%	+12-15%	⚠️ PARTIAL
mid_mt_gap	39.85 M	40.63 M	+0.78 M	+1.97%	+8-10%	❌ FAIL

Comparison 3: Best 3 of 5 (Peak Performance)

Benchmark	Baseline	Phase 6-A	Absolute	Relative	Expected	Result
random_mixed	53.54 M	54.43 M	+0.89 M	+1.67%	+12-15%	❌ FAIL
mid_mt_gap	41.06 M	41.60 M	+0.54 M	+1.33%	+8-10%	❌ FAIL

---

Performance Summary

Overall Results (Best 3 of 5 method)

• random_mixed: 53.54 → 54.43 M ops/s (+1.67%)
• mid_mt_gap: 41.06 → 41.60 M ops/s (+1.33%)

vs Predictions

• random_mixed: Expected +12-15%, Actual +1.67% → FAIL (8-10x smaller than expected)
• mid_mt_gap: Expected +8-10%, Actual +1.33% → FAIL (6-7x smaller than expected)

Interpretation

Phase 6-A shows statistically measurable but practically negligible performance improvements:

• Excluding warmup: +4.07% (random_mixed), +1.97% (mid_mt_gap)
• Best 3 of 5: +1.67% (random_mixed), +1.33% (mid_mt_gap)
• All runs: -2.39% (random_mixed), +1.46% (mid_mt_gap)

The improvements are 8-10x smaller than expected based on perf analysis.

---

Root Cause Analysis

Why the Discrepancy?

The perf profile showed hak_super_lookup() consuming 15.84% of CPU time, yet removing it yields only ~1-4% improvement. Possible explanations:

1. Compiler Optimization (Most Likely)

The compiler may already be optimizing away the hak_super_lookup() call in release builds:

• Dead Store Elimination: The result of hak_super_lookup() is only used for debug logging
• Inlining + Constant Propagation: With LTO, the compiler sees the result is unused
• Evidence: Phase 6-A guard has minimal impact, suggesting code was already "free"

Action: Examine assembly output to verify if hak_super_lookup() is present in baseline build

2. Perf Sampling Bias

The perf profile may have been captured during a different workload phase:

• Different allocation patterns (class distribution)
• Different cache states (cold vs. hot)
• Different thread counts (single vs. multi-threaded)

Action: Re-run perf on the exact benchmark workload to verify 15.84% claim

3. Measurement Noise

The benchmarks show high variance:

• random_mixed: 1.86 M stdev (3.6% of mean)
• mid_mt_gap: 1.65 M stdev (4.1% of mean)

The measured improvements (+1-4%) are within 1-2 standard deviations of noise.

Action: Run longer benchmarks (10M+ operations) to reduce noise

4. Lookup Already Cache-Friendly

The SuperSlab registry lookup may be highly cache-efficient in these workloads:

• Small working set (256 blocks) fits in L1/L2 cache
• Registry entries for active SuperSlabs are hot
• Cost is much lower than perf's 15.84% suggests

Action: Benchmark with larger working sets (4KB+) to stress cache

5. Wrong Hot Path

The perf profile showed 15.84% CPU in hak_super_lookup(), but this may not be on the allocation hot path that these benchmarks exercise:

• The call is in tiny_region_id_write_header() (allocation)
• Benchmarks mix alloc+free, free path may dominate
• Perf may have sampled during a malloc-heavy phase

Action: Isolate allocation-only benchmark (no frees) to verify

---

Recommendations

Immediate Actions

1. HOLD on committing Phase 6-A until investigation completes

   • Current results don't justify the change
   • Risk: code churn without measurable benefit

2. Verify Compiler Behavior

   # Generate assembly for baseline build
   gcc -S -DHAKMEM_BUILD_RELEASE=1 -O3 -o baseline.s core/tiny_region_id.h
   
   # Check if hak_super_lookup appears
   grep "hak_super_lookup" baseline.s
   
   # If absent: compiler already eliminated it (explains minimal improvement)
   # If present: something else is going on
   

3. Re-run Perf on Benchmark Workload

   # Build baseline without Phase 6-A
   git stash
   make clean && make bench_random_mixed_hakmem
   
   # Profile the exact benchmark
   perf record -g ./bench_random_mixed_hakmem 10000000 256 42
   perf report --stdio | grep -A20 "hak_super_lookup"
   
   # Verify if 15.84% claim holds for this workload
   

4. Longer Benchmark Runs

   # 100M operations to reduce noise
   for i in 1 2 3 4 5; do
       ./bench_random_mixed_hakmem 100000000 256 42 2>/dev/null
   done
   

Long-Term Considerations

If investigation reveals:

Scenario A: Compiler Already Optimized

• Decision: Commit Phase 6-A for code cleanliness (no harm, no foul)
• Rationale: Explicitly documents debug-only code, prevents future confusion
• Benefit: Future-proof if compiler behavior changes

Scenario B: Perf Was Wrong

• Decision: Discard Phase 6-A, update perf methodology
• Rationale: The 15.84% CPU claim was based on flawed profiling
• Action: Document correct perf sampling procedure

Scenario C: Benchmark Doesn't Stress Hot Path

• Decision: Commit Phase 6-A, improve benchmark coverage
• Rationale: Real workloads may show the expected gains
• Action: Add allocation-heavy benchmark (e.g., 90% malloc, 10% free)

Scenario D: Measurement Noise Dominates

• Decision: Commit Phase 6-A if longer runs show >5% improvement
• Rationale: Noise can hide real improvements
• Action: Use mimalloc-bench suite for more stable measurements

---

Next Steps

Phase 6-B: Conditional Path Forward

Option 1: Investigate First (Recommended)

1. Run assembly analysis (1 hour)
2. Re-run perf on benchmark (2 hours)
3. Run longer benchmarks (4 hours)
4. Make data-driven decision

Option 2: Commit Anyway

• Rationale: Code is cleaner, no measurable harm
• Risk: Future confusion if optimization isn't actually needed

Option 3: Discard Phase 6-A

• Rationale: No measurable benefit, not worth the churn
• Risk: Miss real optimization if measurement was flawed

---

Appendix: Full Benchmark Output

Baseline - bench_random_mixed

=== Baseline: bench_random_mixed (Before Phase 6-A, SuperSlab lookup ALWAYS enabled) ===
Run 1: Throughput =  53806309 ops/s [iter=1000000 ws=256] time=0.019s
Run 2: Throughput =  53246568 ops/s [iter=1000000 ws=256] time=0.019s
Run 3: Throughput =  53563123 ops/s [iter=1000000 ws=256] time=0.019s
Run 4: Throughput =  49409566 ops/s [iter=1000000 ws=256] time=0.020s
Run 5: Throughput =  51412515 ops/s [iter=1000000 ws=256] time=0.019s

Phase 6-A - bench_random_mixed

=== Phase 6-A: bench_random_mixed (Release build, SuperSlab lookup DISABLED) ===
Run 1: Throughput =  39111201 ops/s [iter=1000000 ws=256] time=0.026s
Run 2: Throughput =  53296242 ops/s [iter=1000000 ws=256] time=0.019s
Run 3: Throughput =  56279982 ops/s [iter=1000000 ws=256] time=0.018s
Run 4: Throughput =  52790754 ops/s [iter=1000000 ws=256] time=0.019s
Run 5: Throughput =  53715992 ops/s [iter=1000000 ws=256] time=0.019s

Baseline - bench_mid_mt_gap

=== Baseline: bench_mid_mt_gap (Before Phase 6-A, SuperSlab lookup ALWAYS enabled) ===
Run 1: Throughput = 41.70 M operations per second, relative time: 0.023979 s.
Run 2: Throughput = 37.39 M operations per second, relative time: 0.026745 s.
Run 3: Throughput = 40.91 M operations per second, relative time: 0.024445 s.
Run 4: Throughput = 40.53 M operations per second, relative time: 0.024671 s.
Run 5: Throughput = 40.56 M operations per second, relative time: 0.024657 s.

Phase 6-A - bench_mid_mt_gap

=== Phase 6-A: bench_mid_mt_gap (Release build, SuperSlab lookup DISABLED) ===
Run 1: Throughput = 41.49 M operations per second, relative time: 0.024103 s.
Run 2: Throughput = 41.81 M operations per second, relative time: 0.023917 s.
Run 3: Throughput = 41.51 M operations per second, relative time: 0.024089 s.
Run 4: Throughput = 38.43 M operations per second, relative time: 0.026019 s.
Run 5: Throughput = 40.78 M operations per second, relative time: 0.024524 s.

---

Conclusion

Phase 6-A successfully implements the intended optimization (disabling SuperSlab lookup in release builds), but the measured performance impact (+1-4%) is 8-10x smaller than the expected +12-15% based on perf analysis.

Critical Question: Why does removing code that perf claims costs 15.84% CPU only yield 1-4% improvement?

Most Likely Answer: The compiler was already optimizing away the hak_super_lookup() call in release builds through dead code elimination, since its result is only used for debug assertions.

Recommended Action: Investigate before committing. If the compiler was already optimizing, Phase 6-A is still valuable for code clarity and future-proofing, but the performance claim needs correction.