hakmem/docs/design/REFACTORING_PLAN_TINY_ALLOC.md

HAKMEM Tiny Allocator Refactoring Plan

Executive Summary

Problem: tiny_alloc_fast() generates 2624 lines of assembly (should be ~20-50 lines for a fast path), causing 11.6x more L1 cache misses than System malloc (1.98 miss/op vs 0.17). Performance: 23.6M ops/s vs System's 92.6M ops/s (3.9x slower).

Root Cause: Architectural bloat from accumulation of experimental features:

• 26 conditional compilation branches in tiny_alloc_fast.inc.h
• 38 runtime conditional checks in allocation path
• 11 overlapping frontend layers (Ring Cache, Unified Cache, HeapV2, UltraHot, FastCache, SFC, etc.)
• 2228-line monolithic hakmem_tiny.c
• 885-line tiny_alloc_fast.inc.h with excessive inlining

Impact: The "smart features" designed to improve performance are creating instruction cache thrashing, destroying the fast path.

---

Analysis: Current Architecture Problems

Problem 1: Too Many Frontend Layers (Bloat Disease)

Current layers in tiny_alloc_fast() (lines 562-812):

static inline void* tiny_alloc_fast(size_t size) {
    // Layer 0: FastCache (C0-C3 only) - lines 232-244
    if (g_fastcache_enable && class_idx <= 3) { ... }

    // Layer 1: SFC (Super Front Cache) - lines 255-274
    if (sfc_is_enabled) { ... }

    // Layer 2: Front C23 (Ultra-simple C2/C3) - lines 610-617
    if (tiny_front_c23_enabled() && (class_idx == 2 || class_idx == 3)) { ... }

    // Layer 3: Unified Cache (tcache-style) - lines 623-635
    if (unified_cache_enabled()) { ... }

    // Layer 4: Ring Cache (C2/C3/C5 only) - lines 641-659
    if (class_idx == 2 || class_idx == 3) { ... }

    // Layer 5: UltraHot (C2-C5) - lines 669-686
    if (ultra_hot_enabled() && front_prune_ultrahot_enabled()) { ... }

    // Layer 6: HeapV2 (C0-C3) - lines 693-701
    if (tiny_heap_v2_enabled() && front_prune_heapv2_enabled() && class_idx <= 3) { ... }

    // Layer 7: Class5 hotpath (256B dedicated) - lines 710-732
    if (hot_c5) { ... }

    // Layer 8: TLS SLL (generic) - lines 736-752
    if (g_tls_sll_enable && !s_front_direct_alloc) { ... }

    // Layer 9: Front-Direct refill - lines 759-775
    if (s_front_direct_alloc) { ... }

    // Layer 10: Legacy refill - lines 769-775
    else { ... }

    // Layer 11: Slow path - lines 806-809
    ptr = hak_tiny_alloc_slow(size, class_idx);
}

Problem: 11 layers with overlapping responsibilities!

• Redundancy: Ring Cache (C2/C3), Front C23 (C2/C3), and UltraHot (C2-C5) all target the same classes
• Branch explosion: Each layer adds 2-5 conditional branches
• I-cache thrashing: 2624 assembly lines cannot fit in L1 instruction cache (32KB = ~10K instructions)

Problem 2: Assembly Bloat Analysis

Expected fast path (System malloc tcache):

; 3-4 instructions, ~10-15 bytes
mov    rax, QWORD PTR [tls_cache + class*8]   ; Load head
test   rax, rax                                 ; Check NULL
je     .miss                                    ; Branch on empty
mov    rdx, QWORD PTR [rax]                    ; Load next
mov    QWORD PTR [tls_cache + class*8], rdx   ; Update head
ret                                             ; Return ptr
.miss:
  call   tcache_refill                          ; Refill (cold path)

Actual HAKMEM fast path: 2624 lines of assembly!

Why?

1. Inlining explosion: Every __attribute__((always_inline)) layer inlines ALL branches
2. ENV checks: Multiple getenv() calls inlined (even with TLS caching)
3. Debug code: Not gated properly with #if !HAKMEM_BUILD_RELEASE
4. Metrics: Frontend metrics tracking (front_metrics_*) adds 50-100 instructions

Problem 3: File Organization Chaos

hakmem_tiny.c (2228 lines):

• Lines 1-500: Global state, TLS variables, initialization
• Lines 500-1000: TLS operations (refill, spill, bind)
• Lines 1000-1500: SuperSlab management
• Lines 1500-2000: Registry operations, slab management
• Lines 2000-2228: Statistics, lifecycle, API wrappers

Problems:

• No clear separation of concerns
• Mix of hot path (refill) and cold path (init, stats)
• Circular dependencies between files via #include

---

Refactoring Plan: 3-Phase Approach

Phase 1: Identify and Remove Dead Features (Priority 1, Quick Win)

Goal: Remove experimental features that are disabled or have negative performance impact.

Actions:

1. Audit ENV flags (1 hour):

   grep -r "getenv.*HAKMEM_TINY" core/ | cut -d: -f2 | sort -u > env_flags.txt
   # Identify which are:
   # - Always disabled (default=0, never used)
   # - Negative performance (A/B test showed regression)
   # - Redundant (overlapping with better features)
   

2. Remove confirmed-dead features (2 hours):

   • UltraHot (Phase 19-4): ENV default OFF, adds 11.7% overhead → DELETE
   • HeapV2 (Phase 13-A): ENV gated, overlaps with Ring Cache → DELETE
   • Front C23: Redundant with Ring Cache → DELETE
   • FastCache: Overlaps with SFC → CONSOLIDATE into SFC

3. Simplify to 3-layer hierarchy (result):

   Layer 0: Unified Cache (tcache-style, all classes C0-C7)
   Layer 1: TLS SLL (unlimited overflow)
   Layer 2: SuperSlab backend (refill source)
   

Expected impact: -30-40% assembly size, +10-15% performance

---

Phase 2: Extract Hot Path to Separate File (Priority 1, Critical)

Goal: Create ultra-simple fast path with zero cold code.

File split:

core/tiny_alloc_fast.inc.h  (885 lines)
  ↓
core/tiny_alloc_ultra.inc.h      (50-100 lines, HOT PATH ONLY)
core/tiny_alloc_refill.inc.h     (200-300 lines, refill logic)
core/tiny_alloc_frontend.inc.h   (300-400 lines, frontend layers)
core/tiny_alloc_metrics.inc.h    (100-150 lines, debug/stats)

tiny_alloc_ultra.inc.h (NEW, ultra-simple):

// Ultra-fast path: 10-20 instructions, no branches except miss
static inline void* tiny_alloc_ultra(int class_idx) {
    // Layer 0: Unified Cache (single TLS array)
    void* ptr = g_unified_cache[class_idx].pop();
    if (__builtin_expect(ptr != NULL, 1)) {
        // Fast hit: 3-4 instructions
        HAK_RET_ALLOC(class_idx, ptr);
    }

    // Layer 1: TLS SLL (overflow)
    ptr = tls_sll_pop(class_idx);
    if (ptr) {
        HAK_RET_ALLOC(class_idx, ptr);
    }

    // Miss: delegate to refill (cold path, out-of-line)
    return tiny_alloc_refill_slow(class_idx);
}

Expected assembly:

tiny_alloc_ultra:
  ; ~15-20 instructions total
  mov    rax, [g_unified_cache + class*8]       ; Load cache head
  test   rax, rax                                 ; Check NULL
  je     .try_sll                                 ; Branch on miss
  mov    rdx, [rax]                              ; Load next
  mov    [g_unified_cache + class*8], rdx       ; Update head
  mov    byte [rax], HEADER_MAGIC | class        ; Write header
  lea    rax, [rax + 1]                          ; USER = BASE + 1
  ret                                             ; Return

.try_sll:
  call   tls_sll_pop                             ; Try TLS SLL
  test   rax, rax
  jne    .sll_hit
  call   tiny_alloc_refill_slow                  ; Cold path (out-of-line)
  ret

.sll_hit:
  mov    byte [rax], HEADER_MAGIC | class
  lea    rax, [rax + 1]
  ret

Expected impact: ~20-30 instructions (from 2624), +200-300% performance

---

Phase 3: Refactor hakmem_tiny.c into Modules (Priority 2, Maintainability)

Goal: Split 2228-line monolith into focused, testable modules.

File structure (new):

core/
├── hakmem_tiny.c               (300-400 lines, main API only)
├── tiny_state.c                (200-300 lines, global state)
├── tiny_tls.c                  (300-400 lines, TLS operations)
├── tiny_superslab.c            (400-500 lines, SuperSlab backend)
├── tiny_registry.c             (200-300 lines, slab registry)
├── tiny_lifecycle.c            (200-300 lines, init/shutdown)
├── tiny_stats.c                (200-300 lines, statistics)
└── tiny_alloc_ultra.inc.h      (50-100 lines, FAST PATH)

Module responsibilities:

1. hakmem_tiny.c (300-400 lines):

   • Public API: hak_tiny_alloc(), hak_tiny_free()
   • Wrapper functions only
   • Include order: tiny_alloc_ultra.inc.h → fast path inline

2. tiny_state.c (200-300 lines):

   • Global variables: g_tiny_pool, g_tls_sll_head[], etc.
   • ENV flag parsing (init-time only)
   • Configuration structures

3. tiny_tls.c (300-400 lines):

   • TLS operations: tls_refill(), tls_spill(), tls_bind()
   • TLS cache management
   • Adaptive sizing logic

4. tiny_superslab.c (400-500 lines):

   • SuperSlab allocation: superslab_refill(), superslab_alloc()
   • Slab metadata management
   • Active block tracking

5. tiny_registry.c (200-300 lines):

   • Slab registry: registry_lookup(), registry_register()
   • Hash table operations
   • Owner slab lookup

6. tiny_lifecycle.c (200-300 lines):

   • Initialization: hak_tiny_init()
   • Shutdown: hak_tiny_shutdown()
   • Prewarm: hak_tiny_prewarm_tls_cache()

7. tiny_stats.c (200-300 lines):

   • Statistics collection
   • Debug counters
   • Metrics printing

Benefits:

• Each file < 500 lines (maintainable)
• Clear dependencies (no circular includes)
• Testable in isolation
• Parallel compilation

---

Priority Order & Estimated Impact

Priority 1: Quick Wins (1-2 days)

Task 1.1: Remove dead features (2 hours)

• Delete UltraHot, HeapV2, Front C23
• Remove ENV checks for disabled features
• Impact: -30% assembly, +10% performance

Task 1.2: Extract ultra-fast path (4 hours)

• Create tiny_alloc_ultra.inc.h (50 lines)
• Move refill logic to separate file
• Impact: -90% assembly (2624 → 200 lines), +150-200% performance

Task 1.3: Remove debug code from release builds (2 hours)

• Gate all fprintf() with #if !HAKMEM_BUILD_RELEASE
• Remove profiling counters in release
• Impact: -10% assembly, +5-10% performance

Expected total (Priority 1): 23.6M → 60-80M ops/s (+150-240%)

---

Priority 2: Code Health (2-3 days)

Task 2.1: Split hakmem_tiny.c (1 day)

• Extract modules as described above
• Fix include dependencies
• Impact: Maintainability only (no performance change)

Task 2.2: Simplify frontend to 2 layers (1 day)

• Unified Cache (Layer 0) + TLS SLL (Layer 1)
• Remove redundant Ring/SFC/FastCache
• Impact: -5-10% assembly, +5-10% performance

Task 2.3: Documentation (0.5 day)

• Document new architecture in ARCHITECTURE.md
• Add performance benchmarks
• Impact: Team velocity +20%

---

Priority 3: Advanced Optimization (3-5 days, optional)

Task 3.1: Profile-guided optimization

• Collect PGO data from benchmarks
• Recompile with -fprofile-use
• Impact: +10-20% performance

Task 3.2: Assembly-level tuning

• Hand-optimize critical sections
• Align hot paths to cache lines
• Impact: +5-10% performance

---

Recommended Implementation Order

Week 1 (Priority 1 - Quick Wins):

1. Day 1: Remove dead features + create tiny_alloc_ultra.inc.h
2. Day 2: Test + benchmark + iterate

Week 2 (Priority 2 - Code Health): 3. Day 3-4: Split hakmem_tiny.c into modules 4. Day 5: Simplify frontend layers

Week 3 (Priority 3 - Optional): 5. Day 6-7: PGO + assembly tuning

---

Expected Performance Results

Current (baseline):

• Performance: 23.6M ops/s
• Assembly: 2624 lines
• L1 misses: 1.98 miss/op

After Priority 1 (Quick Wins):

• Performance: 60-80M ops/s (+150-240%)
• Assembly: 150-200 lines (-92%)
• L1 misses: 0.4-0.6 miss/op (-70%)

After Priority 2 (Code Health):

• Performance: 70-90M ops/s (+200-280%)
• Assembly: 100-150 lines (-94%)
• L1 misses: 0.2-0.4 miss/op (-80%)
• Maintainability: Much improved

Target (System malloc parity):

• Performance: 92.6M ops/s (System malloc baseline)
• Assembly: 50-100 lines (tcache equivalent)
• L1 misses: 0.17 miss/op (System malloc level)

---

Risk Assessment

Low Risk:

• Removing disabled features (UltraHot, HeapV2, Front C23)
• Extracting fast path to separate file
• Gating debug code with #if !HAKMEM_BUILD_RELEASE

Medium Risk:

• Simplifying frontend from 11 layers → 2 layers
  • Mitigation: Keep Ring Cache as fallback during transition
  • A/B test: Toggle via HAKMEM_TINY_UNIFIED_ONLY=1

High Risk:

• Splitting hakmem_tiny.c (circular dependencies)
  • Mitigation: Incremental extraction, one module at a time
  • Test: Ensure all benchmarks pass after each extraction

---

Conclusion

The current architecture suffers from feature accumulation disease: 11 experimental frontend layers competing in the same allocation path, creating massive instruction bloat (2624 lines of assembly). The solution is aggressive simplification:

1. Remove dead/redundant features (11 layers → 2 layers)
2. Extract ultra-fast path (2624 asm lines → 100-150 lines)
3. Split monolithic file (2228 lines → 7 focused modules)

Expected outcome: 3-4x performance improvement (23.6M → 70-90M ops/s), approaching System malloc parity (92.6M ops/s).

Recommended action: Start with Priority 1 tasks (1-2 days), which deliver 80% of the benefit with minimal risk.