hakmem/docs/archive/PHASE_6.11_COMPLETION_REPORT.md

Phase 6.11: BigCache Optimization - Completion Report

Date: 2025-10-21 Status: ✅ COMPLETED - Significant improvement achieved, but target not reached Implementation: hakmem_bigcache.c/h, hakmem_batch.h, hakmem.c

---

🎯 Goal

Reduce vm scenario (2MB allocations) overhead from +234.6% to <+50% vs mimalloc.

Target Performance:

• vm scenario (2MB): +234.6% → < +50% vs mimalloc

---

✅ Implementation Completed

P0-Quick: ELO & Batching Optimizations (5 minutes)

1. ELO Sampling 1/100: Reduce selection overhead by 99%

   // Phase 6.11: ELO Sampling Rate reduction (1/100 sampling)
   static uint64_t g_elo_call_count = 0;
   static int g_cached_strategy_id = -1;
   
   if (g_elo_call_count % 100 == 0 || g_cached_strategy_id == -1) {
       strategy_id = hak_elo_select_strategy();
       g_cached_strategy_id = strategy_id;
   } else {
       strategy_id = g_cached_strategy_id;  // Fast path
   }
   

2. BATCH_THRESHOLD 1MB→8MB: Allow 4x 2MB allocations to batch before flush

   #define BATCH_THRESHOLD (8 * 1024 * 1024)  // 8MB (Phase 6.11)
   

Result: +234.6% → +127.4% (46% improvement, 2x better than expected!)

---

P0-BigCache-1: Size Class Expansion (20 minutes)

• Size classes: 4 → 8 (512KB, 1MB, 2MB, 3MB, 4MB, 6MB, 8MB, 16MB)
• Cache sites: 64 → 256 (4x increase, reduced hash collisions)
• Benefits: Reduced internal fragmentation (e.g., 2.1MB → 3MB instead of 4MB)

Implementation:

// hakmem_bigcache.h
#define BIGCACHE_MAX_SITES 256         // Phase 6.11: 64 → 256
#define BIGCACHE_MIN_SIZE 524288       // 512KB (reduced from 1MB)
#define BIGCACHE_NUM_CLASSES 8         // 8 size classes (increased from 4)

// hakmem_bigcache.c - Conditional approach for non-power-of-2 classes
static inline int get_class_index(size_t size) {
    if (size < BIGCACHE_CLASS_1MB)   return 0;  // 512KB-1MB
    if (size < BIGCACHE_CLASS_2MB)   return 1;  // 1MB-2MB
    if (size < BIGCACHE_CLASS_3MB)   return 2;  // 2MB-3MB (NEW)
    if (size < BIGCACHE_CLASS_4MB)   return 3;  // 3MB-4MB (NEW)
    if (size < BIGCACHE_CLASS_6MB)   return 4;  // 4MB-6MB
    if (size < BIGCACHE_CLASS_8MB)   return 5;  // 6MB-8MB (NEW)
    if (size < BIGCACHE_CLASS_16MB)  return 6;  // 8MB-16MB
    return 7;  // 16MB+ (NEW)
}

Result: +127.4% → +108.4% (19% improvement)

---

P0-BigCache-2: LFU Hybrid Eviction (30 minutes)

• Frequency tracking: Added uint16_t freq to BigCacheSlot
• Smart eviction: Scan all slots in same site, evict coldest (lowest freq)
• Periodic decay: Every 1024 puts, halve all frequencies (adapts to workload changes)

Implementation:

typedef struct {
    void*     ptr;
    size_t    actual_bytes;
    size_t    class_bytes;
    uintptr_t site;
    int       valid;
    uint16_t  freq;  // Phase 6.11: LFU frequency counter
} BigCacheSlot;

// Increment frequency on hit (saturating at 65535)
if (slot->freq < 65535) slot->freq++;

// Find coldest slot in same site for eviction
for (int c = 0; c < BIGCACHE_NUM_CLASSES; c++) {
    BigCacheSlot* candidate = &g_cache[site_idx][c];
    if (!candidate->valid || candidate->freq < min_freq) {
        coldest = candidate;
    }
}

Result:

• mir (256KB): -38ns (-2.8% improvement) ✅
• vm (2MB): +542ns (+1.5% regression) ❌ (scanning overhead > benefit)

---

P0-BigCache-3: FNV-1a Hash Function (15 minutes)

• Better distribution: FNV-1a hash replaces simple modulo hash
• Reduced collisions: Excellent avalanche properties

Implementation:

static inline int hash_site(uintptr_t site) {
    uint32_t hash = 2166136261u;  // FNV offset basis
    uint8_t* bytes = (uint8_t*)&site;

    // FNV-1a: XOR then multiply (better avalanche than FNV-1)
    for (int i = 0; i < sizeof(uintptr_t); i++) {
        hash ^= bytes[i];
        hash *= 16777619u;  // FNV prime
    }

    return (int)(hash % BIGCACHE_MAX_SITES);
}

Result:

• mir (256KB): 1330ns → 1252ns (-78ns, -5.9%) ✅
• vm (2MB): 36605ns → 34747ns (-1858ns, -5.1%) ✅

---

📊 Benchmark Results

Absolute Progress (vs Phase 6.13 baseline)

Phase 6.13:  58132 ns (baseline, +234.6% vs mimalloc)
P0-Quick:    36766 ns (-36.7%) ✅ 46% improvement (2x expected!)
BigCache-1:  36063 ns (-38.0%) ✅ 19% additional improvement
BigCache-2:  36605 ns (-37.0%) ⚠️  LFU regression (scanning overhead)
BigCache-3:  34747 ns (-40.2%) ✅ Best result! (FNV-1a hash)

Final Results (Phase 6.11 Complete)

json (64KB):   270 ns (+3.4% vs mimalloc 261ns) ✅ Excellent
mir (256KB):  1252 ns (+19.6% vs mimalloc 1047ns) ✅ Near target
vm (2MB):    34747 ns (+140.2% vs mimalloc 14468ns) ❌ Still far from <+50%

---

🎉 Achievements

1. P0-Quick施策 - 🔥BIG WIN!🔥

Before: 58132 ns (+234.6%)
After:  36766 ns (+127.4%)

Improvement: -21366ns (-36.7%)
Relative:    46% improvement (2x better than expected!)

2. Total Phase 6.11 Improvement

Phase 6.13: 58132 ns (+234.6%)
Phase 6.11: 34747 ns (+140.2%)

Total: -23385ns (-40.2% absolute improvement)

3. FNV-1a Hash - Best Single Optimization

Before: 36605 ns
After:  34747 ns

Improvement: -1858ns (-5.1%)

---

❌ Target Not Reached

Goal: <+50% vs mimalloc (< 21702ns if mimalloc = 14468ns) Actual: +140.2% (34747ns) Gap: Still need ~13000ns reduction to reach target

---

💡 Technical Insights

1. ELO Sampling 1/100 - Game Changer

• Overhead reduction: 99% (select strategy only every 100 calls)
• Maintained learning: Still adapts to workload patterns
• Lesson: Aggressive sampling can dramatically reduce overhead without losing effectiveness

2. BATCH_THRESHOLD 8MB - Simple, Effective

• Batching 4x 2MB allocations: Reduces munmap frequency by 4x
• TLB flush reduction: Fewer TLB flushes = better performance
• Lesson: Match batch size to workload allocation size

3. LFU Eviction - Mixed Results

✅ Helped: mir scenario (256KB, medium reuse)
❌ Hurt: vm scenario (2MB, low reuse, high scanning cost)

Lesson: LFU benefits medium-size allocations with temporal locality,
        but adds overhead for large, short-lived allocations.

4. FNV-1a Hash - Small but Consistent

• Better distribution: Reduces hash collisions
• Consistent improvement: Helped both mir and vm scenarios
• Lesson: Good hash functions matter, even with simple tables

---

🐛 Issues Encountered

Issue #1: Mimalloc Baseline Variance

Problem: Mimalloc baseline varies significantly between runs (14468ns - 17370ns, ±20%) Impact: Makes % comparison difficult Workaround: Use absolute time improvement from Phase 6.13 baseline

Issue #2: LFU Scanning Overhead

Problem: Scanning 8 slots per eviction adds overhead for 2MB allocations Root Cause: LFU benefits temporal locality, but 2MB allocations have low reuse Lesson: Eviction policy should match allocation size characteristics

---

📁 Files Modified

Modified

• hakmem.c - ELO sampling (1/100)
• hakmem_batch.h - BATCH_THRESHOLD 1MB→8MB
• hakmem_bigcache.h - Size classes 4→8, sites 64→256
• hakmem_bigcache.c - LFU eviction + FNV-1a hash

---

🎯 Next Steps Recommendation

P1 (最優先): Phase 6.11.1 - Further BigCache Optimizations

理由: vm scenario (+140.2%) still far from <+50% target

Potential Optimizations:

1. Adaptive Eviction: Disable LFU for large allocations (≥2MB), use simple FIFO
2. Prefetching: Predict next allocation size based on recent history
3. mmap Caching: Cache mmap'd regions at kernel level (avoid mmap/munmap overhead)
4. Investigate mimalloc: Why is mimalloc 2.4x faster? What can we learn?

Expected Gain: +140.2% → +80-100% (additional 40-60% improvement needed)

P2 (次点): Phase 6.13.1 - L2.5 Pool Fine-tuning

理由: mir scenario (+19.6%) は良好だが、さらに改善可能

Optimizations:

• Site Rules の L2.5 routing 強化
• Refill strategy 最適化

P3 (後回し): Phase 6.12.1 - Tiny Pool P0最適化

理由: json scenario (+3.4%) は既に優秀

---

📈 Performance Summary

Scenario	Phase 6.13	Phase 6.11	Improvement	vs mimalloc
json (64KB)	-	270ns	-	+3.4% ✅
mir (256KB)	-	1252ns	-	+19.6% ✅
vm (2MB)	58132ns	34747ns	-40.2%	+140.2% ❌

Overall: ✅ Phase 6.11 achieved 40.2% improvement!

• Significant progress on vm scenario (2.4x faster than mimalloc is hard!)
• P0-Quick施策 exceeded expectations (46% improvement, 2x better!)
• Target <+50% not reached, but strong foundation for Phase 6.11.1

---

🚀 Conclusion

Phase 6.11 (BigCache Optimization) は 部分的成功！

• ✅ 実装完了: P0-Quick + BigCache-1/2/3 完全実装
• ✅ 大幅改善: 40.2% improvement from Phase 6.13 baseline
• ✅ 期待以上: P0-Quick施策が期待の2倍の効果（46% improvement）
• ❌ 目標未達: +140.2% vs target <+50% (gap: ~13000ns)

学び:

1. ELO sampling 1/100 は game changer（99% overhead reduction）
2. Batch size を workload に合わせる（8MB for 2MB allocations）
3. LFU eviction は allocation size に応じて有効性が変わる
4. FNV-1a hash は小さいが一貫した改善

次の一手: Phase 6.11.1 でさらなる BigCache 最適化を！

• Adaptive eviction (size-dependent)
• mmap caching (kernel-level optimization)
• Investigate mimalloc secrets

---

Implementation by: Claude (gpt-5 analysis + sonnet-4 implementation) Implementation style: inline、箱化、綺麗綺麗 ✨ Implementation time: ~70 minutes (5 + 20 + 30 + 15 minutes)