hakmem/docs/PERF_ANALYSIS_TINY_MIXED.md

Tiny Mixed Workload Performance Analysis

Date: 2025-11-01 Benchmark: bench_random_mixed (200K cycles, 400 ws, seed=1) Size Range: 8-128B (16 size classes) Thread Count: 1 (single-threaded)

Executive Summary

HAKMEM is 32% slower than mimalloc on tiny mixed workloads:

• HAKMEM: 16.46 M ops/sec
• mimalloc: 24.21 M ops/sec
• Relative: 68% of mimalloc speed

Root Cause: HAKMEM spends 3x more CPU cycles in allocator functions (49%) compared to mimalloc (17%).

---

Performance Breakdown

Overall Workload Profile

Both allocators spend ~60% of time in random number generation (libc __random, __random_r), which is expected for this benchmark.

The critical difference is in allocator overhead:

Allocator	Total Overhead	malloc/new	free/delete
mimalloc	17%	7.35%	9.77%
HAKMEM	49%	27%	21.64%

---

HAKMEM Bottlenecks

1. free() Path (21.64% total, 6.75% self)

Perf annotate highlights:

9d45:  pthread_mutex_lock      (2.29%)   ← Global lock on g_mid_registry
9d53:  Binary search loop                ← Search g_mid_registry (Mid range check)
9dad:  pthread_mutex_unlock    (3.93%)   ← Unlock
9db5:  hak_tiny_owner_slab     (4.98%)   ← SECOND ownership check (Tiny range)
9dce:  hak_tiny_free                     ← Actual free

Problems:

1. Double ownership check:
   • First: Global mutex + binary search in g_mid_registry (for Mid range 8KB-32KB)
   • Second: hak_tiny_owner_slab (for Tiny range 8B-1KB)
2. Global mutex contention: Every free() locks g_mid_registry even for Tiny allocations (8-128B)
3. Unnecessary Mid lookup: Mixed workload uses 8-128B sizes, which never hit Mid range

Impact:

• mutex lock/unlock: 6.22%
• hak_pool_mid_lookup (binary search): 7.08%
• hak_tiny_owner_slab: 3.50%
• Total wasted on ownership: ~16.8%

---

2. hak_tiny_alloc_slow Path (9.33% total, 2.95% self)

Perf annotate highlights:

Function prologue (push registers): 14.05%  ← Heavy stack frame
Multi-stage fallback:
  1. hotmag_try_refill (if class <= 3)
  2. TLS list check (g_tls_list_enable)
  3. superslab check (g_use_superslab)
  4. Magazine refill

Problems:

1. Complex fallback chain: 4+ branches before actual allocation
2. TLS variable access overhead: Multiple %fs: prefixed loads
3. Branch misprediction: Conditional checks for hotmag, superslab, etc.

Impact:

• Function overhead: ~3%
• Fallback logic: ~6%

---

3. Allocation Path Summary

Function	Total %	Self %
free	21.64	6.75
hak_tiny_alloc_slow	9.33	2.95
hak_pool_mid_lookup	7.08	2.92
hak_tiny_alloc	7.12	2.81
hak_tiny_owner_slab	3.50	2.41
malloc	3.67	2.41
Total HAKMEM overhead	49%	-

---

mimalloc Efficiency

mimalloc achieves 17% total allocator overhead through:

1. Fast ownership checks: Likely uses pointer bit patterns or alignment tricks (no mutex)
2. Simple allocation path: mi_malloc (7.35%) has minimal branching
3. Lockless thread-local caches: No global mutex for common cases

---

Optimization Roadmap

Priority 1: Eliminate Unnecessary Mid Lookup in free() 🔴

Current flow:

free(ptr)
  ↓
Lock g_mid_registry mutex
  ↓
Binary search g_mid_registry (8KB-32KB check)
  ↓
Unlock g_mid_registry mutex
  ↓
hak_tiny_owner_slab (8B-1KB check)  ← SHOULD BE FIRST!

Proposed flow:

free(ptr)
  ↓
Fast TLS range check (Tiny: 8B-1KB)  ← NO MUTEX
  ↓
  If miss: Check Mid registry (with mutex)
  ↓
  If miss: Check L25/other

Expected gain:

• Eliminate 6.22% (mutex) + 7.08% (mid_lookup) = ~13% speedup
• For Tiny-dominated workloads: ~40% of total overhead removed

---

Priority 2: Streamline free() Ownership Check 🟡

Current approach:

• hak_tiny_owner_slab: Linear search through TLS slabs (3.50% overhead)

Proposed approach (mimalloc-style):

1. Alignment check: if (ptr & (SLAB_SIZE - 1)) == 0 → Not from slab
2. Range check: if (ptr >= tls_heap_start && ptr < tls_heap_end) → TLS-owned
3. Slab header: Direct jump to slab metadata at (ptr & ~(SLAB_SIZE - 1))

Expected gain:

• Reduce hak_tiny_owner_slab from 3.50% → ~0.5%
• Net speedup: ~3%

---

Priority 3: Simplify hak_tiny_alloc_slow Fallback 🟢

Current approach:

• 4-stage fallback: hotmag → TLS list → superslab → magazine

Proposed approach:

1. Check TLS magazine first (most common)
2. Check superslab (if enabled)
3. Refill from central depot

Expected gain:

• Reduce branching overhead: ~2-3% speedup

---

Priority 4: Inline Hot Functions 🟢

Candidates:

• hak_tiny_alloc (7.12% total, 2.81% self) → Inline into malloc
• hak_tiny_owner_slab (3.50% total) → Inline into free

Expected gain:

• Function call overhead reduction: ~2%

---

Comparison with Mid MT Workload

Metric	Tiny Mixed (this)	Mid MT (8KB-32KB)
HAKMEM performance	68% of mimalloc	138% of mimalloc
Size range	8-128B (16 classes)	8KB-32KB (3 classes)
Thread count	1 thread	4 threads
HAKMEM strength	❌ Weak	✅ Strong

Key insight: HAKMEM's Tiny allocator is not optimized for high-frequency small allocations. The Mid MT allocator excels due to:

• Fewer size classes (3 vs 16)
• Lock-free TLS caching
• Optimized for 8KB+ allocations

---

Next Steps

1. ✅ DONE: Profile with perf to identify bottlenecks
2. 🔄 IN PROGRESS: Document findings (this file)
3. ⏳ TODO: Implement Priority 1 optimization (fast TLS range check in free())
4. ⏳ TODO: Benchmark improvement
5. ⏳ TODO: Iterate on Priority 2-4 optimizations

---

Raw Perf Data

HAKMEM Top Functions (>1%)

Children  Self    Samples  Symbol
57.97%    0.00%   0        __libc_start_call_main
43.11%    27.67%  321      __random
28.97%    15.14%  179      __random_r
21.64%    6.75%   79       free
9.33%     2.95%   35       hak_tiny_alloc_slow.constprop.0
7.12%     2.81%   33       hak_tiny_alloc
7.08%     2.92%   35       hak_pool_mid_lookup.constprop.0
3.67%     2.41%   29       malloc
3.50%     2.41%   29       hak_tiny_owner_slab

mimalloc Top Functions (>1%)

Children  Self    Samples  Symbol
99.74%    0.00%   0        __libc_start_call_main
61.43%    41.47%  331      __random
43.63%    22.50%  177      __random_r
29.39%    19.88%  157      main
9.77%     2.42%   19       operator delete[](void*)
7.35%     1.80%   14       mi_malloc

---

Profiling Commands Used

# HAKMEM profiling
perf record -o perf_random_mixed_hakmi.data -F 999 -g ./bench_random_mixed_hakmem 200000 400 1
perf report -i perf_random_mixed_hakmi.data --stdio -n --percent-limit 1

# mimalloc profiling
LD_LIBRARY_PATH=./mimalloc-bench/extern/mi/out/release \
  perf record -o perf_random_mixed_mi.data -F 999 -g ./bench_random_mixed_mi 200000 400 1
perf report -i perf_random_mixed_mi.data --stdio -n --percent-limit 1

# Annotate specific functions
perf annotate -i perf_random_mixed_hakmi.data --stdio free
perf annotate -i perf_random_mixed_hakmi.data --stdio "hak_tiny_alloc_slow.constprop.0"