hakmem/docs/analysis/PHASE52_TAIL_LATENCY_PROXY_RESULTS.md

Phase 52: Tail Latency Proxy Results

Date: 2025-12-16 Phase: 52 - Tail Latency Proxy Measurement Status: COMPLETE (Measurement-only, no code changes)

Executive Summary

We measured tail latency using epoch throughput distribution as a proxy across three allocators:

• hakmem FAST (current optimized build)
• mimalloc (industry baseline)
• system malloc (glibc)

Test configuration: 5-minute single-process soak, 1-second epochs, WS=400 (mixed workload)

Key Findings

1. mimalloc has best tail behavior: Lowest p99/p999 latency proxy, tightest distribution
2. system malloc has second-best tail: Very consistent, low variance
3. hakmem FAST has worst tail: Higher p99/p999, more variability
4. hakmem's gap is in tail consistency, not average performance

Important Note (Method Correction)

Tail の向きと計算には注意が必要:

• Throughput の “tail” は 低い throughput 側（p1/p0.1）を見る（p99 は “速い側”）。
• Latency proxy の percentiles は per-epoch latency（lat_ns = 1e9/throughput）配列を作ってから計算する。
  • p99(latency) != 1e9 / p99(throughput)（非線形 + 順序反転のため）

推奨: CSV（scripts/soak_mixed_single_process.sh 出力）から scripts/analyze_epoch_tail_csv.py で再集計し、SSOT を更新する。

python3 scripts/analyze_epoch_tail_csv.py tail_epoch_hakmem_fast_5m.csv

Detailed Results (v0)

Throughput Distribution (ops/sec)

Metric	hakmem FAST	mimalloc	system malloc
p50	47,887,721	98,738,326	69,562,115
p90	58,629,195	99,580,629	69,931,575
p99	59,174,766	110,702,822	70,165,415
p999	59,567,912	111,190,037	70,308,452
Mean	50,174,657	99,084,977	69,447,599
Std Dev	4,461,290	2,455,894	522,021
Min	46,254,013	95,458,811	66,242,568
Max	59,608,715	111,202,228	70,326,858

Latency Proxy (ns/op)

Calculated as 1 / throughput * 1e9 to convert throughput to per-operation latency.

Metric	hakmem FAST	mimalloc	system malloc
p50	20.88 ns	10.13 ns	14.38 ns
p90	21.12 ns	10.24 ns	14.50 ns
p99	21.33 ns	10.43 ns	14.80 ns
p999	21.57 ns	10.47 ns	15.07 ns
Mean	20.07 ns	10.10 ns	14.40 ns
Std Dev	1.60 ns	0.23 ns	0.11 ns
Min	16.78 ns	8.99 ns	14.22 ns
Max	21.62 ns	10.48 ns	15.10 ns

Analysis

Tail Behavior Comparison

Standard Deviation as % of Mean (lower = more consistent):

• hakmem FAST: 7.98% (highest variability)
• mimalloc: 2.28% (good consistency)
• system malloc: 0.77% (best consistency)

p99/p50 Ratio (lower = better tail):

• hakmem FAST: 1.024 (2.4% tail slowdown)
• mimalloc: 1.030 (3.0% tail slowdown)
• system malloc: 1.029 (2.9% tail slowdown)

p999/p50 Ratio:

• hakmem FAST: 1.033 (3.3% tail slowdown)
• mimalloc: 1.034 (3.4% tail slowdown)
• system malloc: 1.048 (4.8% tail slowdown)

Interpretation

1. hakmem's throughput variance is high: 4.46M ops/sec std dev vs mimalloc's 2.46M and system's 0.52M

   • This indicates periodic slowdowns or stalls
   • Likely due to TLS cache misses, metadata lookup costs, or GC-like background work

2. mimalloc has best absolute performance AND good tail behavior:

   • 2x faster than hakmem at median
   • Lower latency at all percentiles
   • Moderate variance (2.28% std dev)

3. system malloc has rock-solid consistency:

   • Only 0.77% std dev (extremely stable)
   • Very tight p99/p999 spread
   • Middle performance tier (~1.5x faster than hakmem)

4. hakmem's tail problem is relative to its mean:

   • Absolute p99 latency (21.33 ns) isn't terrible
   • But variance is 2-3x higher than competitors
   • Suggests optimization opportunities in cache warmth, metadata layout

Implications for Optimization

Root Causes to Investigate

1. TLS cache thrashing: High variance suggests periodic cache coldness
2. Metadata lookup cost: Possibly slower on cache misses
3. Background work interference: Adaptive sizing, stats collection?
4. Free path delays: Remote frees, mailbox processing

Potential Solutions

1. Prewarm more aggressively: Reduce cold-start penalties
2. Optimize metadata cache hit rate: Better locality, prefetching
3. Reduce background work frequency: Less interruption to hot path
4. Improve free-side batching: Reduce per-operation variance

Prioritization

Given that:

• hakmem is already 2x slower than mimalloc at median
• Tail behavior is worse but not catastrophically so
• Variance is the main issue, not worst-case absolute latency

Recommendation: Focus on reducing variance rather than chasing p999 specifically.

• Target: Get std dev down from 4.46M to <2M ops/sec (match mimalloc's 2.46M)
• This will naturally improve tail latency as a side effect

Test Configuration

Hardware

• CPU: (recorded in soak CSV metadata)
• Memory: Sufficient for WS=400 (20MB prefault)
• OS: Linux

Benchmark Parameters

• Workload: bench_random_mixed (70% malloc, 30% free)
• Working Set: 400 (mixed size distribution)
• Duration: 300 seconds (5 minutes)
• Epoch Length: 1 second
• Process Model: Single process (no parallelism)

Allocator Builds

• hakmem: MINIMAL build (FAST path enabled, aggressive inlining)
• mimalloc: Default build from vendor
• system malloc: glibc default (no LD_PRELOAD)

Raw Data

CSV files available at:

• /mnt/workdisk/public_share/hakmem/tail_epoch_hakmem_fast_5m.csv
• /mnt/workdisk/public_share/hakmem/tail_epoch_mimalloc_5m.csv
• /mnt/workdisk/public_share/hakmem/tail_epoch_system_5m.csv

Analysis script: scripts/calculate_percentiles.py

Next Steps

1. Phase 53: RSS Tax Triage - understand memory overhead
2. Future optimization phases: Target variance reduction
   • Phase 54+: TLS cache optimization
   • Phase 55+: Metadata locality improvements
   • Phase 56+: Background work reduction

Conclusion

Phase 52 Status: COMPLETE

We have established a tail latency baseline using epoch throughput as a proxy. Key takeaway: hakmem's tail behavior is acceptable but has room for improvement, primarily by reducing throughput variance (std dev). This measurement provides a clear target for future optimization work.

No code changes made - this was a measurement-only phase.