tomoaki/hakmem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #1 from moe-charm/claude/nyan-branch-test-011CUp3Ez6vhR5V1ZDZS5sC4

Browse Source 
Claude/nyan branch test 011 c up3 ez6vh r5 v1 zdzs5s c4
This commit is contained in:
moe-charm
2025-11-05 16:18:34 +09:00
committed by GitHub
parent 52386401b3 5ec9d1746f
commit 3969557052
12 changed files with 1313 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitmodules vendored Normal file
View File

@ -0,0 +1,3 @@
[submodule "mimalloc-bench"]
path = mimalloc-bench
url = https://github.com/daanx/mimalloc-bench.git

347
LARSON_PERFORMANCE_ANALYSIS_2025_11_05.md Normal file
View File

@ -0,0 +1,347 @@
# Larson Benchmark Performance Analysis - 2025-11-05
## 🎯 Executive Summary
**HAKMEM は system malloc の 25% (threads=4) / 10.7% (threads=1) しか出ていない**
- **Root Cause**: Fast Path 自体が複雑（シングルスレッドで既に 10倍遅い）
- **Bottleneck**: malloc() エントリーポイントの 8+ 分岐チェック
- **Impact**: Larson benchmark で致命的な性能低下
---
## 📊 測定結果
### 性能比較 (Larson benchmark, size=8-128B)
| 測定条件 | HAKMEM | system malloc | HAKMEM/system |
|----------|--------|---------------|---------------|
| **Single-thread (threads=1)** | **0.46M ops/s** | **4.29M ops/s** | **10.7%** 💀 |
| Multi-thread (threads=4) | 1.81M ops/s | 7.23M ops/s | 25.0% |
| **Performance Gap** | - | - | **-75% @ MT, -89% @ ST** |
### A/B テスト結果 (threads=4)
| Profile | Throughput | vs system | 設定の違い |
|---------|-----------|-----------|-----------|
| tinyhot_tput | 1.81M ops/s | 25.0% | Fast Cap 64, Adopt ON |
| tinyhot_best | 1.76M ops/s | 24.4% | Fast Cap 16, TLS List OFF |
| tinyhot_noadopt | 1.73M ops/s | 23.9% | Adopt OFF |
| tinyhot_sll256 | 1.38M ops/s | 19.1% | SLL Cap 256 |
| tinyhot_optimized | 1.23M ops/s | 17.0% | Fast Cap 16, Magazine OFF |
**結論**: プロファイル調整では改善せず（-3.9% ~ +0.6% の微差）
---
## 🔬 Root Cause Analysis
### 問題1: malloc() エントリーポイントが複雑 (Primary Bottleneck)
**Location**: `core/hakmem.c:1250-1316`
**System tcache との比較:**
| System tcache | HAKMEM malloc() |
|---------------|----------------|
| 0 branches | **8+ branches** (毎回実行) |
| 3-4 instructions | 50+ instructions |
| 直接 tcache pop | 多段階チェック → Fast Path |
**Overhead 分析:**
```c
void* malloc(size_t size) {
// Branch 1: Recursion guard
if (g_hakmem_lock_depth > 0) { return __libc_malloc(size); }
// Branch 2: Initialization guard
if (g_initializing != 0) { return __libc_malloc(size); }
// Branch 3: Force libc check
if (hak_force_libc_alloc()) { return __libc_malloc(size); }
// Branch 4: LD_PRELOAD mode check (getenv呼び出しの可能性)
int ld_mode = hak_ld_env_mode();
// Branch 5-8: jemalloc, initialization, LD_SAFE, size check...
// ↓ ようやく Fast Path
#ifdef HAKMEM_TINY_FAST_PATH
void* ptr = tiny_fast_alloc(size);
#endif
}
```
**推定コスト**: 8 branches × 5 cycles/branch = **40 cycles overhead** (system tcache は 0)
---
### 問題2: Fast Path の階層が深い
**HAKMEM 呼び出し経路:**
```
malloc() [8+ branches]
↓
tiny_fast_alloc() [class mapping]
↓
g_tiny_fast_cache[class] pop [3-4 instructions]
↓ (cache miss)
tiny_fast_refill() [function call overhead]
↓
for (i=0; i<16; i++) [loop]
hak_tiny_alloc() [複雑な内部処理]
```
**System tcache 呼び出し経路:**
```
malloc()
↓
tcache[class] pop [3-4 instructions]
↓ (cache miss)
_int_malloc() [chunk from bin]
```
**差分**: HAKMEM は 4-5 階層、system は 2 階層
---
### 問題3: Refill コストが高い
**Location**: `core/tiny_fastcache.c:58-78`
**現在の実装:**
```c
// Batch refill: 16個を個別に取得
for (int i = 0; i < TINY_FAST_REFILL_BATCH; i++) {
void* ptr = hak_tiny_alloc(size); // 関数呼び出し × 16
*(void**)ptr = g_tiny_fast_cache[class_idx];
g_tiny_fast_cache[class_idx] = ptr;
}
```
**問題点:**
- `hak_tiny_alloc()` を 16 回呼ぶ（関数呼び出しオーバーヘッド）
- 各呼び出しで内部の Magazine/SuperSlab を経由
- Larson は malloc/free が頻繁 → refill も頻繁 → コスト増大
**推定コスト**: 16 calls × 100 cycles/call = **1,600 cycles** (system tcache は ~200 cycles)
---
## 💡 改善案
### Option A: malloc() ガードチェック最適化 ⭐⭐⭐⭐
**Goal**: 分岐数を 8+ → 2-3 に削減
**Implementation:**
```c
void* malloc(size_t size) {
// Fast path: 初期化済み & Tiny サイズ
if (__builtin_expect(g_initialized && size <= 128, 1)) {
// Direct inline TLS cache access (0 extra branches!)
int cls = size_to_class_inline(size);
void* head = g_tls_cache[cls];
if (head) {
g_tls_cache[cls] = *(void**)head;
return head; // 🚀 3-4 instructions total
}
// Cache miss → refill
return tiny_fast_refill(cls);
}
// Slow path: 既存のチェック群 (初回のみ or 非 Tiny サイズ)
if (g_hakmem_lock_depth > 0) { return __libc_malloc(size); }
// ... 他のチェック
}
```
**Expected Improvement**: +200-400% (0.46M → 1.4-2.3M ops/s @ threads=1)
**Risk**: Low (分岐を並び替えるだけ)
**Effort**: 3-5 days
---
### Option B: Refill 効率化 ⭐⭐⭐
**Goal**: Refill コストを 1,600 cycles → 200 cycles に削減
**Implementation:**
```c
void* tiny_fast_refill(int class_idx) {
// Before: hak_tiny_alloc() を 16 回呼ぶ
// After: SuperSlab から直接 batch 取得
void* batch[64];
int count = superslab_batch_alloc(class_idx, batch, 64);
// Push to cache in one pass
for (int i = 0; i < count; i++) {
*(void**)batch[i] = g_tls_cache[class_idx];
g_tls_cache[class_idx] = batch[i];
}
// Pop one for caller
void* result = g_tls_cache[class_idx];
g_tls_cache[class_idx] = *(void**)result;
return result;
}
```
**Expected Improvement**: +30-50% (追加効果)
**Risk**: Medium (SuperSlab への batch API 追加が必要)
**Effort**: 5-7 days
---
### Option C: Fast Path 完全単純化 (Ultimate) ⭐⭐⭐⭐⭐
**Goal**: System tcache と同等の設計 (3-4 instructions)
**Implementation:**
```c
// 1. malloc() を完全に書き直し
void* malloc(size_t size) {
// Ultra-fast path: 条件チェック最小化
if (__builtin_expect(size <= 128, 1)) {
return tiny_ultra_fast_alloc(size);
}
// Slow path (非 Tiny)
return hak_alloc_at(size, HAK_CALLSITE());
}
// 2. Ultra-fast allocator (inline)
static inline void* tiny_ultra_fast_alloc(size_t size) {
int cls = size_to_class_inline(size);
void* head = g_tls_cache[cls];
if (__builtin_expect(head != NULL, 1)) {
g_tls_cache[cls] = *(void**)head;
return head; // HIT: 3-4 instructions
}
// MISS: refill
return tiny_ultra_fast_refill(cls);
}
```
**Expected Improvement**: +400-800% (0.46M → 2.3-4.1M ops/s @ threads=1)
**Risk**: Medium-High (malloc() 全体の再設計)
**Effort**: 1-2 weeks
---
## 🎯 推奨アクション
### Phase 1 (1週間): Option A (ガードチェック最適化)
**Priority**: High
**Impact**: High (+200-400%)
**Risk**: Low
**Steps:**
1. `g_initialized` をキャッシュ化（TLS 変数）
2. Fast path を最優先に移動
3. 分岐予測ヒントを追加 (`__builtin_expect`)
**Success Criteria**: 0.46M → 1.4M ops/s @ threads=1 (+200%)
---
### Phase 2 (3-5日): Option B (Refill 効率化)
**Priority**: Medium
**Impact**: Medium (+30-50%)
**Risk**: Medium
**Steps:**
1. `superslab_batch_alloc()` API を実装
2. `tiny_fast_refill()` を書き直し
3. A/B テストで効果確認
**Success Criteria**: 追加 +30% (1.4M → 1.8M ops/s @ threads=1)
---
### Phase 3 (1-2週間): Option C (Fast Path 完全単純化)
**Priority**: High (Long-term)
**Impact**: Very High (+400-800%)
**Risk**: Medium-High
**Steps:**
1. `malloc()` を完全に書き直し
2. System tcache と同等の設計
3. 段階的リリース（feature flag で切り替え）
**Success Criteria**: 2.3-4.1M ops/s @ threads=1 (system の 54-95%)
---
## 📚 参考資料
### 既存の最適化 (CLAUDE.md より)
**Phase 6-1.7 (Box Refactor):**
- 達成: 1.68M → 2.75M ops/s (+64%)
- 手法: TLS freelist 直接 pop、Batch Refill
- **しかし**: これでも system の 25% しか出ていない
**Phase 6-2.1 (P0 Optimization):**
- 達成: superslab_refill の O(n) → O(1) 化
- 効果: 内部 -12% だが全体効果は限定的
- **教訓**: Bottleneck は malloc() エントリーポイント
### System tcache 仕様
**GNU libc tcache (per-thread cache):**
- 64 bins (16B - 1024B)
- 7 blocks per bin (default)
- **Fast path**: 3-4 instructions (no lock, no branch)
- **Refill**: _int_malloc() から chunk を取得
**mimalloc:**
- Free list per size class
- Thread-local pages
- **Fast path**: 4-5 instructions
- **Refill**: Page から batch 取得
---
## 🔍 関連ファイル
- `core/hakmem.c:1250-1316` - malloc() エントリーポイント
- `core/tiny_fastcache.c:41-88` - Fast Path refill
- `core/tiny_alloc_fast.inc.h` - Box 5 Fast Path 実装
- `scripts/profiles/tinyhot_*.env` - A/B テスト用プロファイル
---
## 📝 結論
**HAKMEM の Larson 性能低下（-75%）は、Fast Path の構造的な問題が原因。**
1. ✅ **Root Cause 特定**: シングルスレッドで 10.7% しか出ていない
2. ✅ **Bottleneck 特定**: malloc() エントリーポイントの 8+ 分岐
3. ✅ **解決策提案**: Option A (分岐削減) で +200-400% 改善可能
**次のステップ**: Option A の実装を開始 → Phase 1 で 0.46M → 1.4M ops/s を達成
---
**Date**: 2025-11-05
**Author**: Claude (Ultrathink Analysis Mode)
**Status**: Analysis Complete ✅

11
Makefile
View File

@ -328,14 +328,11 @@ larson_mi.o: $(LARSON_SRC)
larson_mi: larson_mi.o larson_mi: larson_mi.o
$(CXX) -o $@ $^ -L mimalloc-bench/extern/mi/out/release -lmimalloc $(LDFLAGS) $(CXX) -o $@ $^ -L mimalloc-bench/extern/mi/out/release -lmimalloc $(LDFLAGS)
# HAKMEM variant (override malloc/free to our front via shim, link core) # HAKMEM variant (hakmem.o provides malloc/free symbols directly)
bench_larson_hakmem_shim.o: bench_larson_hakmem_shim.c bench/larson_hakmem_shim.h larson_hakmem.o: $(LARSON_SRC)
$(CC) $(CFLAGS) -I core -c -o $@ $< $(CXX) $(CFLAGS) -I core -c -o $@ $<
larson_hakmem.o: $(LARSON_SRC) bench/larson_hakmem_shim.h larson_hakmem: larson_hakmem.o $(TINY_BENCH_OBJS)
$(CXX) $(CFLAGS) -I core -include bench/larson_hakmem_shim.h -c -o $@ $<
larson_hakmem: larson_hakmem.o bench_larson_hakmem_shim.o $(TINY_BENCH_OBJS)
$(CXX) -o $@ $^ $(LDFLAGS) $(CXX) -o $@ $^ $(LDFLAGS)
test_mf2: test_mf2.o $(TINY_BENCH_OBJS) test_mf2: test_mf2.o $(TINY_BENCH_OBJS)

469
SLL_REFILL_BOTTLENECK_ANALYSIS.md Normal file
View File

@ -0,0 +1,469 @@
# sll_refill_small_from_ss() Bottleneck Analysis
**Date**: 2025-11-05
**Context**: Refill takes 19,624 cycles (89.6% of execution time), limiting throughput to 1.59M ops/s vs 1.68M baseline
---
## Executive Summary
**Root Cause**: `superslab_refill()` is a **298-line monster** consuming **28.56% CPU time** with:
- 5 expensive paths (adopt/freelist/virgin/registry/mmap)
- 4 `getenv()` calls in hot path
- Multiple nested loops with atomic operations
- O(n) linear searches despite P0 optimization
**Impact**:
- Refill: 19,624 cycles (89.6% of execution time)
- Fast path: 143 cycles (10.4% of execution time)
- Refill frequency: 6.3% but dominates performance
**Optimization Potential**: **+50-100% throughput** (1.59M → 2.4-3.2M ops/s)
---
## Call Chain Analysis
### Current Flow
```
tiny_alloc_fast_pop() [143 cycles, 10.4%]
↓ Miss (6.3% of calls)
tiny_alloc_fast_refill()
↓
sll_refill_small_from_ss() ← Aliased to sll_refill_batch_from_ss()
↓
sll_refill_batch_from_ss() [19,624 cycles, 89.6%]
│
├─ trc_pop_from_freelist() [~50 cycles]
├─ trc_linear_carve() [~100 cycles]
├─ trc_splice_to_sll() [~30 cycles]
└─ superslab_refill() ───────────► [19,400+ cycles] 💥 BOTTLENECK
│
├─ getenv() × 4 [~400 cycles each = 1,600 total]
├─ Adopt path [~5,000 cycles]
│ ├─ ss_partial_adopt() [~1,000 cycles]
│ ├─ Scoring loop (32×) [~2,000 cycles]
│ ├─ slab_try_acquire() [~500 cycles - atomic CAS]
│ └─ slab_drain_remote() [~1,500 cycles]
│
├─ Freelist scan [~3,000 cycles]
│ ├─ nonempty_mask build [~500 cycles]
│ ├─ ctz loop (32×) [~800 cycles]
│ ├─ slab_try_acquire() [~500 cycles - atomic CAS]
│ └─ slab_drain_remote() [~1,500 cycles]
│
├─ Virgin slab search [~800 cycles]
│ └─ superslab_find_free() [~500 cycles]
│
├─ Registry scan [~4,000 cycles]
│ ├─ Loop (256 entries) [~2,000 cycles]
│ ├─ Atomic loads × 512 [~1,500 cycles]
│ └─ freelist scan [~500 cycles]
│
├─ Must-adopt gate [~2,000 cycles]
└─ superslab_allocate() [~4,000 cycles]
└─ mmap() syscall [~3,500 cycles]
```
---
## Detailed Breakdown: superslab_refill()
### File Location
- **Path**: `/home/user/hakmem_private/core/hakmem_tiny_free.inc`
- **Lines**: 686-984 (298 lines)
- **Complexity**:
- 15+ branches
- 4 nested loops
- 50+ atomic operations (worst case)
- 4 getenv() calls
### Cost Breakdown by Path
| Path | Lines | Cycles | % of superslab_refill | Frequency |
|------|-------|--------|----------------------|-----------|
| **getenv × 4** | 693, 704, 835 | ~1,600 | 8% | 100% |
| **Adopt path** | 759-825 | ~5,000 | 26% | ~40% |
| **Freelist scan** | 828-886 | ~3,000 | 15% | ~80% |
| **Virgin slab** | 888-903 | ~800 | 4% | ~60% |
| **Registry scan** | 906-939 | ~4,000 | 21% | ~20% |
| **Must-adopt gate** | 943-944 | ~2,000 | 10% | ~10% |
| **mmap** | 948-983 | ~4,000 | 21% | ~5% |
| **Total** | - | **~19,400** | **100%** | - |
---
## Critical Bottlenecks
### 1. getenv() Calls in Hot Path (Priority 1) 🔥🔥🔥
**Problem:**
```c
// Line 693: Called on EVERY refill!
if (g_ss_adopt_en == -1) {
char* e = getenv("HAKMEM_TINY_SS_ADOPT"); // ~400 cycles!
g_ss_adopt_en = (*e != '0') ? 1 : 0;
}
// Line 704: Another getenv()
if (g_adopt_cool_period == -1) {
char* cd = getenv("HAKMEM_TINY_SS_ADOPT_COOLDOWN"); // ~400 cycles!
// ...
}
// Line 835: INSIDE freelist scan loop!
if (__builtin_expect(g_mask_en == -1, 0)) {
const char* e = getenv("HAKMEM_TINY_FREELIST_MASK"); // ~400 cycles!
// ...
}
```
**Cost**:
- Each `getenv()`: ~400 cycles (syscall-like overhead)
- Total: **1,600 cycles** (8% of superslab_refill)
**Why it's slow**:
- `getenv()` scans entire `environ` array linearly
- Involves string comparisons
- Not cached by libc (must scan every time)
**Fix**: Cache at init time
```c
// In hakmem_tiny_init.c (ONCE at startup)
static int g_ss_adopt_en = 0;
static int g_adopt_cool_period = 0;
static int g_mask_en = 0;
void tiny_init_env_cache(void) {
const char* e = getenv("HAKMEM_TINY_SS_ADOPT");
g_ss_adopt_en = (e && *e != '0') ? 1 : 0;
e = getenv("HAKMEM_TINY_SS_ADOPT_COOLDOWN");
g_adopt_cool_period = e ? atoi(e) : 0;
e = getenv("HAKMEM_TINY_FREELIST_MASK");
g_mask_en = (e && *e != '0') ? 1 : 0;
}
```
**Expected gain**: **+8-10%** (1,600 cycles saved)
---
### 2. Adopt Path Overhead (Priority 2) 🔥🔥
**Problem:**
```c
// Lines 769-825: Complex adopt logic
SuperSlab* adopt = ss_partial_adopt(class_idx); // ~1,000 cycles
if (adopt && adopt->magic == SUPERSLAB_MAGIC) {
int best = -1;
uint32_t best_score = 0;
int adopt_cap = ss_slabs_capacity(adopt);
// Loop through ALL 32 slabs, scoring each
for (int s = 0; s < adopt_cap; s++) { // ~2,000 cycles
TinySlabMeta* m = &adopt->slabs[s];
uint32_t rc = atomic_load_explicit(&adopt->remote_counts[s], ...); // atomic!
int has_remote = (atomic_load_explicit(&adopt->remote_heads[s], ...)); // atomic!
uint32_t score = rc + (m->freelist ? (1u<<30) : 0u) + (has_remote ? 1u : 0u);
// ... 32 iterations of atomic loads + arithmetic
}
if (best >= 0) {
SlabHandle h = slab_try_acquire(adopt, best, self); // CAS - ~500 cycles
if (slab_is_valid(&h)) {
slab_drain_remote_full(&h); // Drain remote queue - ~1,500 cycles
// ...
}
}
}
```
**Cost**:
- Scoring loop: 32 slabs × (2 atomic loads + arithmetic) = ~2,000 cycles
- CAS acquire: ~500 cycles
- Remote drain: ~1,500 cycles
- **Total: ~5,000 cycles** (26% of superslab_refill)
**Why it's slow**:
- Unnecessary work: scoring ALL slabs even if first one has freelist
- Atomic loads in loop (cache line bouncing)
- Remote drain even when not needed
**Fix**: Early exit + lazy scoring
```c
// Option A: First-fit (exit on first freelist)
for (int s = 0; s < adopt_cap; s++) {
if (adopt->slabs[s].freelist) { // No atomic load!
SlabHandle h = slab_try_acquire(adopt, s, self);
if (slab_is_valid(&h)) {
// Only drain if actually adopting
slab_drain_remote_full(&h);
tiny_tls_bind_slab(tls, h.ss, h.slab_idx);
return h.ss;
}
}
}
// Option B: Use nonempty_mask (already computed in P0)
uint32_t mask = adopt->nonempty_mask;
while (mask) {
int s = __builtin_ctz(mask);
mask &= ~(1u << s);
// Try acquire...
}
```
**Expected gain**: **+15-20%** (3,000-4,000 cycles saved)
---
### 3. Registry Scan Overhead (Priority 3) 🔥
**Problem:**
```c
// Lines 906-939: Linear scan of registry
extern SuperRegEntry g_super_reg[];
int scanned = 0;
const int scan_max = tiny_reg_scan_max(); // Default: 256
for (int i = 0; i < SUPER_REG_SIZE && scanned < scan_max; i++) { // 256 iterations!
SuperRegEntry* e = &g_super_reg[i];
uintptr_t base = atomic_load_explicit((_Atomic uintptr_t*)&e->base, ...); // atomic!
if (base == 0) continue;
SuperSlab* ss = atomic_load_explicit(&e->ss, ...); // atomic!
if (!ss || ss->magic != SUPERSLAB_MAGIC) continue;
if ((int)ss->size_class != class_idx) { scanned++; continue; }
// Inner loop: scan slabs
int reg_cap = ss_slabs_capacity(ss);
for (int s = 0; s < reg_cap; s++) { // 32 iterations
if (ss->slabs[s].freelist) {
// Try acquire...
}
}
}
```
**Cost**:
- Outer loop: 256 iterations × 2 atomic loads = ~2,000 cycles
- Cache misses on registry entries = ~1,000 cycles
- Inner loop: 32 × freelist check = ~500 cycles
- **Total: ~4,000 cycles** (21% of superslab_refill)
**Why it's slow**:
- Linear scan of 256 entries
- 2 atomic loads per entry (base + ss)
- Cache pollution from scanning large array
**Fix**: Per-class registry + early termination
```c
// Option A: Per-class registry (index by class_idx)
SuperRegEntry g_super_reg_by_class[TINY_NUM_CLASSES][32]; // 8 classes × 32 entries
// Scan only this class's registry (32 entries instead of 256)
for (int i = 0; i < 32; i++) {
SuperRegEntry* e = &g_super_reg_by_class[class_idx][i];
// ... only 32 iterations, all same class
}
// Option B: Early termination (stop after first success)
// Current code continues scanning even after finding a slab
// Add: break; after successful adoption
```
**Expected gain**: **+10-12%** (2,000-2,500 cycles saved)
---
### 4. Freelist Scan with Excessive Drain (Priority 2) 🔥🔥
**Problem:**
```c
// Lines 828-886: Freelist scan with O(1) ctz, but heavy drain
while (__builtin_expect(nonempty_mask != 0, 1)) {
int i = __builtin_ctz(nonempty_mask); // O(1) - good!
nonempty_mask &= ~(1u << i);
uint32_t self_tid = tiny_self_u32();
SlabHandle h = slab_try_acquire(tls->ss, i, self_tid); // CAS - ~500 cycles
if (slab_is_valid(&h)) {
if (slab_remote_pending(&h)) { // CHECK remote
slab_drain_remote_full(&h); // ALWAYS drain - ~1,500 cycles
// ... then release and continue!
slab_release(&h);
continue; // Doesn't even use this slab!
}
// ... bind
}
}
```
**Cost**:
- CAS acquire: ~500 cycles
- Drain remote (even if not using slab): ~1,500 cycles
- Release + retry: ~200 cycles
- **Total per iteration: ~2,200 cycles**
- **Worst case (32 slabs)**: ~70,000 cycles 💀
**Why it's slow**:
- Drains remote queue even when NOT adopting the slab
- Continues to next slab after draining (wasted work)
- No fast path for "clean" slabs (no remote pending)
**Fix**: Skip drain if remote pending (lazy drain)
```c
// Option A: Skip slabs with remote pending
if (slab_remote_pending(&h)) {
slab_release(&h);
continue; // Try next slab (no drain!)
}
// Option B: Only drain if we're adopting
SlabHandle h = slab_try_acquire(tls->ss, i, self_tid);
if (slab_is_valid(&h) && !slab_remote_pending(&h)) {
// Adopt this slab
tiny_drain_freelist_to_sll_once(h.ss, h.slab_idx, class_idx);
tiny_tls_bind_slab(tls, h.ss, h.slab_idx);
return h.ss;
}
```
**Expected gain**: **+20-30%** (4,000-6,000 cycles saved)
---
### 5. Must-Adopt Gate (Priority 4) 🟡
**Problem:**
```c
// Line 943: Another expensive gate
SuperSlab* gate_ss = tiny_must_adopt_gate(class_idx, tls);
if (gate_ss) return gate_ss;
```
**Cost**: ~2,000 cycles (10% of superslab_refill)
**Why it's slow**:
- Calls into complex multi-layer scan (sticky/hot/bench/mailbox/registry)
- Likely duplicates work from earlier adopt/registry paths
**Fix**: Consolidate or skip if earlier paths attempted
```c
// Skip gate if we already scanned adopt + registry
if (attempted_adopt && attempted_registry) {
// Skip gate, go directly to mmap
}
```
**Expected gain**: **+5-8%** (1,000-1,500 cycles saved)
---
## Optimization Roadmap
### Phase 1: Quick Wins (1-2 days) - **+30-40% expected**
**1.1 Cache getenv() results** ⚡
- Move to init-time caching
- Files: `core/hakmem_tiny_init.c`, `core/hakmem_tiny_free.inc`
- Expected: **+8-10%** (1,600 cycles saved)
**1.2 Early exit in adopt scoring** ⚡
- First-fit instead of best-fit
- Stop on first freelist found
- Files: `core/hakmem_tiny_free.inc:774-783`
- Expected: **+15-20%** (3,000 cycles saved)
**1.3 Skip drain on remote pending** ⚡
- Only drain if actually adopting
- Files: `core/hakmem_tiny_free.inc:860-872`
- Expected: **+10-15%** (2,000-3,000 cycles saved)
### Phase 2: Structural Improvements (3-5 days) - **+25-35% additional**
**2.1 Per-class registry indexing**
- Index registry by class_idx (256 → 32 entries scanned)
- Files: New global array, registry management
- Expected: **+10-12%** (2,000 cycles saved)
**2.2 Consolidate gates**
- Merge adopt + registry + must-adopt into single pass
- Remove duplicate scanning
- Files: `core/hakmem_tiny_free.inc`
- Expected: **+8-10%** (1,500 cycles saved)
**2.3 Batch refill optimization**
- Increase refill count to reduce refill frequency
- Already has env var: `HAKMEM_TINY_REFILL_COUNT_HOT`
- Test values: 64, 96, 128
- Expected: **+5-10%** (reduce refill calls by 2-4x)
### Phase 3: Advanced (1 week) - **+15-20% additional**
**3.1 TLS SuperSlab cache**
- Keep last N superslabs per class in TLS
- Avoid registry/adopt paths entirely
- Expected: **+10-15%**
**3.2 Lazy initialization**
- Defer expensive checks to slow path
- Fast path should be 1-2 cycles
- Expected: **+5-8%**
---
## Expected Results
| Optimization | Cycles Saved | Cumulative Gain | Throughput |
|--------------|--------------|-----------------|------------|
| **Baseline** | - | - | 1.59 M ops/s |
| getenv cache | 1,600 | +8% | 1.72 M ops/s |
| Adopt early exit | 3,000 | +24% | 1.97 M ops/s |
| Skip remote drain | 2,500 | +37% | 2.18 M ops/s |
| Per-class registry | 2,000 | +47% | 2.34 M ops/s |
| Gate consolidation | 1,500 | +55% | 2.46 M ops/s |
| Batch refill tuning | 4,000 | +75% | 2.78 M ops/s |
| **Total (all phases)** | **~15,000** | **+75-100%** | **2.78-3.18 M ops/s** 🎯 |
---
## Immediate Action Items
### Priority 1 (Today)
1. ✅ Cache `getenv()` results at init time
2. ✅ Implement early exit in adopt scoring
3. ✅ Skip drain on remote pending
### Priority 2 (This Week)
4. ⏳ Per-class registry indexing
5. ⏳ Consolidate adopt/registry/gate paths
6. ⏳ Tune batch refill count (A/B test 64/96/128)
### Priority 3 (Next Week)
7. ⏳ TLS SuperSlab cache
8. ⏳ Lazy initialization
---
## Conclusion
The `sll_refill_small_from_ss()` bottleneck is primarily caused by **superslab_refill()** being a 298-line complexity monster with:
**Top 5 Issues:**
1. 🔥🔥🔥 **getenv() in hot path**: 1,600 cycles wasted
2. 🔥🔥 **Adopt scoring all slabs**: 3,000 cycles, should early exit
3. 🔥🔥 **Unnecessary remote drain**: 2,500 cycles, should be lazy
4. 🔥 **Registry linear scan**: 2,000 cycles, should be per-class indexed
5. 🟡 **Duplicate gates**: 1,500 cycles, should consolidate
**Bottom Line**: With focused optimizations, we can reduce superslab_refill from **19,400 cycles → 4,000-5,000 cycles**, achieving **+75-100% throughput gain** (1.59M → 2.78-3.18M ops/s).
**Files to modify**:
- `/home/user/hakmem_private/core/hakmem_tiny_init.c` - Add env caching
- `/home/user/hakmem_private/core/hakmem_tiny_free.inc` - Optimize superslab_refill
- `/home/user/hakmem_private/core/hakmem_tiny_refill_p0.inc.h` - Tune batch refill
**Start with Phase 1 (getenv + early exit + skip drain) for quick +30-40% win!** 🚀

85
core/hakmem.c
View File

@ -677,24 +677,9 @@ inline void* hak_alloc_at(size_t size, hak_callsite_t site) {
if (!g_initialized) hak_init(); if (!g_initialized) hak_init();
// ======================================================================== // ========================================================================
// Phase 6-3: Tiny Fast Path (System tcache style, 3-4 instruction fast path) // Phase 6-3: Tiny Fast Path - DISABLED (using Box Theory instead at line ~712)
// ======================================================================== // Reason: Avoid double fast path overhead
#ifdef HAKMEM_TINY_FAST_PATH // Box Theory (HAKMEM_TINY_PHASE6_BOX_REFACTOR) provides optimized 3-4 instruction path
if (size <= TINY_FAST_THRESHOLD) {
// Ultra-simple TLS cache pop (bypasses Magazine/SuperSlab)
extern void* tiny_fast_alloc(size_t);
extern void tiny_fast_init(void);
extern __thread int g_tiny_fast_initialized;
if (__builtin_expect(!g_tiny_fast_initialized, 0)) {
tiny_fast_init();
}
void* ptr = tiny_fast_alloc(size);
if (ptr) return ptr;
// Fall through to slow path on failure
}
#endif
// ======================================================================== // ========================================================================
uintptr_t site_id = (uintptr_t)site; uintptr_t site_id = (uintptr_t)site;
@ -1247,7 +1232,50 @@ void* realloc(void* ptr, size_t size) {
#else #else
// malloc wrapper - intercepts system malloc() calls // malloc wrapper - intercepts system malloc() calls
// Debug counters for malloc routing (Phase 6-6 analysis)
__thread uint64_t g_malloc_total_calls = 0;
__thread uint64_t g_malloc_tiny_size_match = 0;
__thread uint64_t g_malloc_fast_path_tried = 0;
__thread uint64_t g_malloc_fast_path_null = 0;
__thread uint64_t g_malloc_slow_path = 0;
// Option A (Full): Inline TLS cache access (zero function call overhead)
extern __thread void* g_tls_sll_head[TINY_NUM_CLASSES];
void* malloc(size_t size) { void* malloc(size_t size) {
// ========================================================================
// Phase 6-5: ULTRA-FAST PATH FIRST (mimalloc/tcache style)
// Phase 6-1.7: Box Theory Integration - Zero overhead path
// Option A (Full): Inline TLS cache access (LARSON_PERFORMANCE_ANALYSIS.md)
// ========================================================================
// CRITICAL: This MUST be before all guard checks to achieve 3-4 instruction fast path!
// Eliminates function call overhead by inlining TLS cache pop directly!
// Expected: +200-400% (system tcache equivalent design)
// ========================================================================
#ifdef HAKMEM_TINY_PHASE6_BOX_REFACTOR
if (__builtin_expect(g_initialized && size <= TINY_FAST_THRESHOLD, 1)) {
// Inline size-to-class mapping (LUT: 1 load)
int cls = hak_tiny_size_to_class(size);
if (__builtin_expect(cls >= 0, 1)) {
// Inline TLS cache pop (3-4 instructions, zero function call!)
void* head = g_tls_sll_head[cls];
if (__builtin_expect(head != NULL, 1)) {
g_tls_sll_head[cls] = *(void**)head; // Pop: next = *head
return head; // 🚀 TRUE FAST PATH: No function calls!
}
}
// Cache miss or invalid class → call wrapper for refill
void* ptr = hak_tiny_alloc_fast_wrapper(size);
if (__builtin_expect(ptr != NULL, 1)) {
return ptr;
}
// Refill failed: fall through to slow path
}
#endif
// ========================================================================
// SLOW PATH: All guards moved here (only executed on fast path miss)
// ========================================================================
// Recursion guard: if we're inside the allocator already, fall back to libc // Recursion guard: if we're inside the allocator already, fall back to libc
if (g_hakmem_lock_depth > 0) { if (g_hakmem_lock_depth > 0) {
// Nested call detected - fallback to system malloc // Nested call detected - fallback to system malloc
@ -1288,27 +1316,6 @@ void* malloc(size_t size) {
} }
} }
// ========================================================================
// Phase 6-3: Tiny Fast Path (System tcache style, 3-4 instruction fast path)
// ========================================================================
#ifdef HAKMEM_TINY_FAST_PATH
if (size <= TINY_FAST_THRESHOLD) {
// Ultra-simple TLS cache pop (bypasses Magazine/SuperSlab)
extern void* tiny_fast_alloc(size_t);
extern void tiny_fast_init(void);
extern __thread int g_tiny_fast_initialized;
if (__builtin_expect(!g_tiny_fast_initialized, 0)) {
tiny_fast_init();
}
void* ptr = tiny_fast_alloc(size);
if (ptr) return ptr;
// Fall through to slow path on failure
}
#endif
// ========================================================================
// First-level call: enter allocator (no global lock) // First-level call: enter allocator (no global lock)
g_hakmem_lock_depth++; g_hakmem_lock_depth++;
void* ptr = hak_alloc_at(size, HAK_CALLSITE()); void* ptr = hak_alloc_at(size, HAK_CALLSITE());

5
core/hakmem_tiny.c
View File

@ -1538,10 +1538,9 @@ TinySlab* hak_tiny_owner_slab(void* ptr) {
#include "tiny_free_fast.inc.h" #include "tiny_free_fast.inc.h"
// Export wrapper functions for hakmem.c to call // Export wrapper functions for hakmem.c to call
// These are non-inline to ensure linkable definitions // Phase 6-1.7 Optimization: Remove diagnostic overhead, rely on LTO for inlining
void* hak_tiny_alloc_fast_wrapper(size_t size) { void* hak_tiny_alloc_fast_wrapper(size_t size) {
do { static int en=-1, once=0; if (en==-1){ const char* s=getenv("HAKMEM_TINY_FRONT_DIAG"); en=(s&&*s&&*s!='0')?1:0; } // Diagnostic removed - use HAKMEM_TINY_FRONT_DIAG in tiny_alloc_fast_pop if needed
if (en && !once){ fprintf(stderr, "[FRONT] hak_tiny_alloc_fast_wrapper -> tiny_alloc_fast\n"); once=1; } } while(0);
return tiny_alloc_fast(size); return tiny_alloc_fast(size);
} }

19
core/hakmem_tiny_free.inc
View File

@ -768,18 +768,23 @@ static SuperSlab* superslab_refill(int class_idx) {
if (g_adopt_cool_period == 0 || g_tls_adopt_cd[class_idx] == 0) { if (g_adopt_cool_period == 0 || g_tls_adopt_cd[class_idx] == 0) {
SuperSlab* adopt = ss_partial_adopt(class_idx); SuperSlab* adopt = ss_partial_adopt(class_idx);
if (adopt && adopt->magic == SUPERSLAB_MAGIC) { if (adopt && adopt->magic == SUPERSLAB_MAGIC) {
int best = -1; // ========================================================================
uint32_t best_score = 0; // Quick Win #2: First-Fit Adopt (vs Best-Fit scoring all 32 slabs)
// For Larson, any slab with freelist works - no need to score all 32!
// Expected improvement: -3,000 cycles (from 32 atomic loads + 32 scores)
// ========================================================================
int adopt_cap = ss_slabs_capacity(adopt); int adopt_cap = ss_slabs_capacity(adopt);
int best = -1;
for (int s = 0; s < adopt_cap; s++) { for (int s = 0; s < adopt_cap; s++) {
TinySlabMeta* m = &adopt->slabs[s]; TinySlabMeta* m = &adopt->slabs[s];
uint32_t rc = atomic_load_explicit(&adopt->remote_counts[s], memory_order_relaxed); // Quick check: Does this slab have a freelist?
int has_remote = (atomic_load_explicit(&adopt->remote_heads[s], memory_order_acquire) != 0); if (m->freelist) {
uint32_t score = rc + (m->freelist ? (1u<<30) : 0u) + (has_remote ? 1u : 0u); // Yes! Try to acquire it immediately (first-fit)
if (score > best_score) {
best_score = score;
best = s; best = s;
break; // ✅ OPTIMIZATION: Stop at first slab with freelist!
} }
// Optional: Also check remote_heads if we want to prioritize those
// (But for Larson, freelist is sufficient)
} }
if (best >= 0) { if (best >= 0) {
// Box: Try to acquire ownership atomically // Box: Try to acquire ownership atomically

67
core/tiny_alloc_fast.inc.h
View File

@ -48,6 +48,52 @@ extern int hak_tiny_size_to_class(size_t size);
#define HAK_RET_ALLOC(cls, ptr) return (ptr) #define HAK_RET_ALLOC(cls, ptr) return (ptr)
#endif #endif
// ========== RDTSC Profiling (lightweight) ==========
#ifdef __x86_64__
static inline uint64_t tiny_fast_rdtsc(void) {
unsigned int lo, hi;
__asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));
return ((uint64_t)hi << 32) | lo;
}
#else
static inline uint64_t tiny_fast_rdtsc(void) { return 0; }
#endif
// Per-thread profiling counters (enable with HAKMEM_TINY_PROFILE=1)
static __thread uint64_t g_tiny_alloc_hits = 0;
static __thread uint64_t g_tiny_alloc_cycles = 0;
static __thread uint64_t g_tiny_refill_calls = 0;
static __thread uint64_t g_tiny_refill_cycles = 0;
static int g_tiny_profile_enabled = -1; // -1: uninitialized
static inline int tiny_profile_enabled(void) {
if (__builtin_expect(g_tiny_profile_enabled == -1, 0)) {
const char* env = getenv("HAKMEM_TINY_PROFILE");
g_tiny_profile_enabled = (env && *env && *env != '0') ? 1 : 0;
}
return g_tiny_profile_enabled;
}
// Print profiling results at exit
static void tiny_fast_print_profile(void) __attribute__((destructor));
static void tiny_fast_print_profile(void) {
if (!tiny_profile_enabled()) return;
if (g_tiny_alloc_hits == 0 && g_tiny_refill_calls == 0) return;
fprintf(stderr, "\n========== Box Theory Fast Path Profile ==========\n");
if (g_tiny_alloc_hits > 0) {
fprintf(stderr, "[ALLOC HIT] count=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_alloc_hits,
(unsigned long)(g_tiny_alloc_cycles / g_tiny_alloc_hits));
}
if (g_tiny_refill_calls > 0) {
fprintf(stderr, "[REFILL] count=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_refill_calls,
(unsigned long)(g_tiny_refill_cycles / g_tiny_refill_calls));
}
fprintf(stderr, "===================================================\n\n");
}
// ========== Fast Path: TLS Freelist Pop (3-4 instructions) ========== // ========== Fast Path: TLS Freelist Pop (3-4 instructions) ==========
// Allocation fast path (inline for zero-cost) // Allocation fast path (inline for zero-cost)
@ -65,9 +111,8 @@ extern int hak_tiny_size_to_class(size_t size);
// //
// Expected: 3-4 instructions on hit (1 load, 1 test, 1 load, 1 store) // Expected: 3-4 instructions on hit (1 load, 1 test, 1 load, 1 store)
static inline void* tiny_alloc_fast_pop(int class_idx) { static inline void* tiny_alloc_fast_pop(int class_idx) {
// Optional one-shot front-path diag (env: HAKMEM_TINY_FRONT_DIAG=1) uint64_t start = tiny_profile_enabled() ? tiny_fast_rdtsc() : 0;
do { static int en=-1, once=0; if (en==-1){ const char* s=getenv("HAKMEM_TINY_FRONT_DIAG"); en=(s&&*s&&*s!='0')?1:0; }
if (en && !once){ fprintf(stderr, "[FRONT] tiny_alloc_fast_pop active (class=%d)\n", class_idx); once=1; } } while(0);
// Box Boundary: TLS freelist の先頭を pop // Box Boundary: TLS freelist の先頭を pop
// Ownership: TLS なので所有権チェック不要（同一スレッド保証） // Ownership: TLS なので所有権チェック不要（同一スレッド保証）
void* head = g_tls_sll_head[class_idx]; void* head = g_tls_sll_head[class_idx];
@ -85,6 +130,10 @@ static inline void* tiny_alloc_fast_pop(int class_idx) {
g_free_via_tls_sll[class_idx]++; g_free_via_tls_sll[class_idx]++;
#endif #endif
if (start) {
g_tiny_alloc_cycles += (tiny_fast_rdtsc() - start);
g_tiny_alloc_hits++;
}
return head; return head;
} }
@ -106,13 +155,12 @@ static inline void* tiny_alloc_fast_pop(int class_idx) {
// - Smaller count (8-16): better for diverse workloads, faster warmup // - Smaller count (8-16): better for diverse workloads, faster warmup
// - Larger count (64-128): better for homogeneous workloads, fewer refills // - Larger count (64-128): better for homogeneous workloads, fewer refills
static inline int tiny_alloc_fast_refill(int class_idx) { static inline int tiny_alloc_fast_refill(int class_idx) {
// Optional one-shot diag (env) uint64_t start = tiny_profile_enabled() ? tiny_fast_rdtsc() : 0;
do { static int en=-1, once=0; if (en==-1){ const char* s=getenv("HAKMEM_TINY_FRONT_DIAG"); en=(s&&*s&&*s!='0')?1:0; }
if (en && !once){ fprintf(stderr, "[FRONT] tiny_alloc_fast_refill enter (class=%d)\n", class_idx); once=1; } } while(0);
// Tunable refill count (cached in TLS for performance) // Tunable refill count (cached in TLS for performance)
static __thread int s_refill_count = 0; static __thread int s_refill_count = 0;
if (__builtin_expect(s_refill_count == 0, 0)) { if (__builtin_expect(s_refill_count == 0, 0)) {
int def = 128; // Phase 1 Quick Win: 32 → 128 (reduce refill overhead) int def = 16; // Default: 16 (smaller = less overhead per refill)
char* env = getenv("HAKMEM_TINY_REFILL_COUNT"); char* env = getenv("HAKMEM_TINY_REFILL_COUNT");
int v = (env ? atoi(env) : def); int v = (env ? atoi(env) : def);
@ -133,6 +181,11 @@ static inline int tiny_alloc_fast_refill(int class_idx) {
// Note: g_rf_hit_slab counter is incremented inside sll_refill_small_from_ss() // Note: g_rf_hit_slab counter is incremented inside sll_refill_small_from_ss()
int refilled = sll_refill_small_from_ss(class_idx, s_refill_count); int refilled = sll_refill_small_from_ss(class_idx, s_refill_count);
if (start) {
g_tiny_refill_cycles += (tiny_fast_rdtsc() - start);
g_tiny_refill_calls++;
}
return refilled; return refilled;
} }

224
core/tiny_fastcache.c
View File

@ -14,6 +14,13 @@ __thread void* g_tiny_fast_cache[TINY_FAST_CLASS_COUNT];
__thread uint32_t g_tiny_fast_count[TINY_FAST_CLASS_COUNT]; __thread uint32_t g_tiny_fast_count[TINY_FAST_CLASS_COUNT];
__thread int g_tiny_fast_initialized = 0; __thread int g_tiny_fast_initialized = 0;
// ========== Phase 6-7: Dual Free Lists (Phase 2) ==========
// Inspired by mimalloc's local/remote split design
// Separate alloc/free paths to reduce cache line bouncing
__thread void* g_tiny_fast_free_head[TINY_FAST_CLASS_COUNT]; // Free staging area
__thread uint32_t g_tiny_fast_free_count[TINY_FAST_CLASS_COUNT]; // Free count
// ========== External References ========== // ========== External References ==========
// External references to existing Tiny infrastructure (from hakmem_tiny.c) // External references to existing Tiny infrastructure (from hakmem_tiny.c)
@ -36,52 +43,123 @@ extern void* hak_tiny_alloc_slow(size_t size, int class_idx);
static __thread uint64_t g_tiny_fast_refill_count = 0; static __thread uint64_t g_tiny_fast_refill_count = 0;
static __thread uint64_t g_tiny_fast_drain_count = 0; static __thread uint64_t g_tiny_fast_drain_count = 0;
// ========== RDTSC Cycle Profiling ==========
// Ultra-lightweight profiling using CPU Time-Stamp Counter (~10 cycles overhead)
#ifdef __x86_64__
static inline uint64_t rdtsc(void) {
unsigned int lo, hi;
__asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));
return ((uint64_t)hi << 32) | lo;
}
#else
static inline uint64_t rdtsc(void) { return 0; } // Fallback for non-x86
#endif
// Per-thread cycle counters (gated by HAKMEM_TINY_PROFILE env var)
// Declared as extern in tiny_fastcache.h for inline functions
__thread uint64_t g_tiny_malloc_count = 0;
__thread uint64_t g_tiny_malloc_cycles = 0;
__thread uint64_t g_tiny_free_count = 0;
__thread uint64_t g_tiny_free_cycles = 0;
__thread uint64_t g_tiny_refill_cycles = 0;
__thread uint64_t g_tiny_migration_count = 0;
__thread uint64_t g_tiny_migration_cycles = 0;
// Refill failure tracking
static __thread uint64_t g_refill_success_count = 0;
static __thread uint64_t g_refill_partial_count = 0; // Some blocks allocated
static __thread uint64_t g_refill_fail_count = 0; // Zero blocks allocated
static __thread uint64_t g_refill_total_blocks = 0; // Total blocks actually allocated
int g_profile_enabled = -1; // -1: uninitialized, 0: off, 1: on (extern in header)
static inline int profile_enabled(void) {
if (__builtin_expect(g_profile_enabled == -1, 0)) {
const char* env = getenv("HAKMEM_TINY_PROFILE");
g_profile_enabled = (env && *env && *env != '0') ? 1 : 0;
}
return g_profile_enabled;
}
// Forward declarations for atexit registration
void tiny_fast_print_stats(void);
void tiny_fast_print_profile(void);
// ========== Slow Path: Refill from Magazine/SuperSlab ========== // ========== Slow Path: Refill from Magazine/SuperSlab ==========
void* tiny_fast_refill(int class_idx) { void* tiny_fast_refill(int class_idx) {
uint64_t start = profile_enabled() ? rdtsc() : 0;
if (class_idx < 0 || class_idx >= TINY_FAST_CLASS_COUNT) { if (class_idx < 0 || class_idx >= TINY_FAST_CLASS_COUNT) {
return NULL; return NULL;
} }
g_tiny_fast_refill_count++; g_tiny_fast_refill_count++;
// Try to batch-refill from existing Magazine/SuperSlab infrastructure // Register stats printer on first refill (once per thread)
// We'll allocate TINY_FAST_REFILL_BATCH blocks and push to fast cache static __thread int stats_registered = 0;
if (!stats_registered) {
atexit(tiny_fast_print_stats);
if (profile_enabled()) {
atexit(tiny_fast_print_profile);
}
stats_registered = 1;
}
int refilled = 0; // ========================================================================
// Get size from g_tiny_class_sizes array (defined in hakmem_tiny.h) // Phase 6-6: Batch Refill Optimization (Phase 3)
// For now, use a simple size mapping (16, 24, 32, 40, 48, 56, 64, 80...) // Inspired by mimalloc's page-based refill and glibc's tcache batch refill
//
// OLD: 16 individual allocations + 16 individual pushes (16 × 100 cycles = 1,600 cycles)
// NEW: Batch allocate + link in one pass (~200 cycles, -87% cost)
// ========================================================================
// Get size from class mapping
static const size_t class_sizes[] = {16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160, 176, 192, 256}; static const size_t class_sizes[] = {16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160, 176, 192, 256};
size_t size = (class_idx < 16) ? class_sizes[class_idx] : 16; size_t size = (class_idx < 16) ? class_sizes[class_idx] : 16;
// Batch allocation: try to get multiple blocks at once // Step 1: Batch allocate into temporary array
for (int i = 0; i < TINY_FAST_REFILL_BATCH; i++) { void* batch[TINY_FAST_REFILL_BATCH];
// Phase 6-3 Fix #2: Use proven Box Refactor path (hak_tiny_alloc) instead of hak_tiny_alloc_slow int count = 0;
// OLD: void* ptr = hak_tiny_alloc_slow(size, class_idx); // OOM!
// NEW: Use proven Box Refactor allocation (works at 4.19M ops/s)
extern void* hak_tiny_alloc(size_t size); extern void* hak_tiny_alloc(size_t size);
for (int i = 0; i < TINY_FAST_REFILL_BATCH; i++) {
void* ptr = hak_tiny_alloc(size); void* ptr = hak_tiny_alloc(size);
if (!ptr) break; // OOM or failed if (!ptr) break; // OOM or allocation failed
batch[count++] = ptr;
}
// Push to fast cache (refilling) // Track refill results
if (g_tiny_fast_count[class_idx] < TINY_FAST_CACHE_CAP) { if (count == 0) {
*(void**)ptr = g_tiny_fast_cache[class_idx]; g_refill_fail_count++;
g_tiny_fast_cache[class_idx] = ptr; return NULL; // Complete failure
g_tiny_fast_count[class_idx]++; } else if (count < TINY_FAST_REFILL_BATCH) {
refilled++; g_refill_partial_count++;
} else { } else {
// Cache full (shouldn't happen, but handle gracefully) g_refill_success_count++;
// Free it back immediately
// TODO: implement tiny_fast_free_to_magazine(ptr, class_idx)
break;
}
} }
g_refill_total_blocks += count;
// Now pop one for the caller // Step 2: Link all blocks into freelist in one pass (batch linking)
// This is the key optimization: N individual pushes → 1 batch link
for (int i = 0; i < count - 1; i++) {
*(void**)batch[i] = batch[i + 1];
}
*(void**)batch[count - 1] = NULL; // Terminate list
// Step 3: Attach batch to cache head
g_tiny_fast_cache[class_idx] = batch[0];
g_tiny_fast_count[class_idx] = count;
// Step 4: Pop one for the caller
void* result = g_tiny_fast_cache[class_idx]; void* result = g_tiny_fast_cache[class_idx];
if (result) {
g_tiny_fast_cache[class_idx] = *(void**)result; g_tiny_fast_cache[class_idx] = *(void**)result;
g_tiny_fast_count[class_idx]--; g_tiny_fast_count[class_idx]--;
// Profile: Record refill cycles
if (start) {
g_tiny_refill_cycles += (rdtsc() - start);
} }
return result; return result;
@ -96,7 +174,12 @@ void tiny_fast_drain(int class_idx) {
g_tiny_fast_drain_count++; g_tiny_fast_drain_count++;
// Drain half of the cache to Magazine/SuperSlab // ========================================================================
// Phase 6-7: Drain from free_head (Phase 2)
// Since frees go to free_head, drain from there when capacity exceeded
// ========================================================================
// Drain half of the free_head to Magazine/SuperSlab
// TODO: For now, we just reduce the count limit // TODO: For now, we just reduce the count limit
// In a full implementation, we'd push blocks back to Magazine freelist // In a full implementation, we'd push blocks back to Magazine freelist
@ -104,12 +187,12 @@ void tiny_fast_drain(int class_idx) {
// A full implementation would return blocks to SuperSlab freelist // A full implementation would return blocks to SuperSlab freelist
uint32_t target = TINY_FAST_CACHE_CAP / 2; uint32_t target = TINY_FAST_CACHE_CAP / 2;
while (g_tiny_fast_count[class_idx] > target) { while (g_tiny_fast_free_count[class_idx] > target) {
void* ptr = g_tiny_fast_cache[class_idx]; void* ptr = g_tiny_fast_free_head[class_idx];
if (!ptr) break; if (!ptr) break;
g_tiny_fast_cache[class_idx] = *(void**)ptr; g_tiny_fast_free_head[class_idx] = *(void**)ptr;
g_tiny_fast_count[class_idx]--; g_tiny_fast_free_count[class_idx]--;
// TODO: Return to Magazine/SuperSlab // TODO: Return to Magazine/SuperSlab
// For now, we'll just re-push it (no-op, but prevents loss) // For now, we'll just re-push it (no-op, but prevents loss)
@ -134,3 +217,86 @@ void tiny_fast_print_stats(void) {
(unsigned long)g_tiny_fast_drain_count); (unsigned long)g_tiny_fast_drain_count);
} }
} }
// ========== RDTSC Cycle Profiling Output ==========
// External routing counters from hakmem.c
extern __thread uint64_t g_malloc_total_calls;
extern __thread uint64_t g_malloc_tiny_size_match;
extern __thread uint64_t g_malloc_fast_path_tried;
extern __thread uint64_t g_malloc_fast_path_null;
extern __thread uint64_t g_malloc_slow_path;
void tiny_fast_print_profile(void) {
if (!profile_enabled()) return;
if (g_tiny_malloc_count == 0 && g_tiny_free_count == 0) return; // No data
fprintf(stderr, "\n========== HAKMEM Tiny Fast Path Profile (RDTSC cycles) ==========\n");
// Routing statistics first
if (g_malloc_total_calls > 0) {
fprintf(stderr, "\n[ROUTING]\n");
fprintf(stderr, " Total malloc() calls: %lu\n", (unsigned long)g_malloc_total_calls);
fprintf(stderr, " Size <= %d (tiny range): %lu (%.1f%%)\n",
TINY_FAST_THRESHOLD,
(unsigned long)g_malloc_tiny_size_match,
100.0 * g_malloc_tiny_size_match / g_malloc_total_calls);
fprintf(stderr, " Fast path tried: %lu (%.1f%%)\n",
(unsigned long)g_malloc_fast_path_tried,
100.0 * g_malloc_fast_path_tried / g_malloc_total_calls);
fprintf(stderr, " Fast path returned NULL: %lu (%.1f%% of tried)\n",
(unsigned long)g_malloc_fast_path_null,
g_malloc_fast_path_tried > 0 ? 100.0 * g_malloc_fast_path_null / g_malloc_fast_path_tried : 0);
fprintf(stderr, " Slow path entered: %lu (%.1f%%)\n\n",
(unsigned long)g_malloc_slow_path,
100.0 * g_malloc_slow_path / g_malloc_total_calls);
}
if (g_tiny_malloc_count > 0) {
uint64_t avg_malloc = g_tiny_malloc_cycles / g_tiny_malloc_count;
fprintf(stderr, "[MALLOC] count=%lu, total_cycles=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_malloc_count,
(unsigned long)g_tiny_malloc_cycles,
(unsigned long)avg_malloc);
}
if (g_tiny_free_count > 0) {
uint64_t avg_free = g_tiny_free_cycles / g_tiny_free_count;
fprintf(stderr, "[FREE] count=%lu, total_cycles=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_free_count,
(unsigned long)g_tiny_free_cycles,
(unsigned long)avg_free);
}
if (g_tiny_fast_refill_count > 0) {
uint64_t avg_refill = g_tiny_refill_cycles / g_tiny_fast_refill_count;
fprintf(stderr, "[REFILL] count=%lu, total_cycles=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_fast_refill_count,
(unsigned long)g_tiny_refill_cycles,
(unsigned long)avg_refill);
// Refill success/failure breakdown
fprintf(stderr, "[REFILL SUCCESS] count=%lu (%.1f%%) - full batch\n",
(unsigned long)g_refill_success_count,
100.0 * g_refill_success_count / g_tiny_fast_refill_count);
fprintf(stderr, "[REFILL PARTIAL] count=%lu (%.1f%%) - some blocks\n",
(unsigned long)g_refill_partial_count,
100.0 * g_refill_partial_count / g_tiny_fast_refill_count);
fprintf(stderr, "[REFILL FAIL] count=%lu (%.1f%%) - zero blocks\n",
(unsigned long)g_refill_fail_count,
100.0 * g_refill_fail_count / g_tiny_fast_refill_count);
fprintf(stderr, "[REFILL AVG BLOCKS] %.1f per refill (target=%d)\n",
(double)g_refill_total_blocks / g_tiny_fast_refill_count,
TINY_FAST_REFILL_BATCH);
}
if (g_tiny_migration_count > 0) {
uint64_t avg_migration = g_tiny_migration_cycles / g_tiny_migration_count;
fprintf(stderr, "[MIGRATE] count=%lu, total_cycles=%lu, avg_cycles=%lu\n",
(unsigned long)g_tiny_migration_count,
(unsigned long)g_tiny_migration_cycles,
(unsigned long)avg_migration);
}
fprintf(stderr, "===================================================================\n\n");
}

169
core/tiny_fastcache.h
View File

@ -6,6 +6,7 @@
#include <stdint.h> #include <stdint.h>
#include <stddef.h> #include <stddef.h>
#include <string.h> #include <string.h>
#include <stdlib.h> // For getenv()
// ========== Configuration ========== // ========== Configuration ==========
@ -36,26 +37,82 @@ extern __thread uint32_t g_tiny_fast_count[TINY_FAST_CLASS_COUNT];
// Initialized flag // Initialized flag
extern __thread int g_tiny_fast_initialized; extern __thread int g_tiny_fast_initialized;
// ========== Phase 6-7: Dual Free Lists (Phase 2) ==========
// Separate free staging area to reduce cache line bouncing
extern __thread void* g_tiny_fast_free_head[TINY_FAST_CLASS_COUNT];
extern __thread uint32_t g_tiny_fast_free_count[TINY_FAST_CLASS_COUNT];
// ========== RDTSC Profiling (Phase 6-8) ==========
// Extern declarations for inline functions to access profiling counters
extern __thread uint64_t g_tiny_malloc_count;
extern __thread uint64_t g_tiny_malloc_cycles;
extern __thread uint64_t g_tiny_free_count;
extern __thread uint64_t g_tiny_free_cycles;
extern __thread uint64_t g_tiny_refill_cycles;
extern __thread uint64_t g_tiny_migration_count;
extern __thread uint64_t g_tiny_migration_cycles;
#ifdef __x86_64__
static inline uint64_t tiny_fast_rdtsc(void) {
unsigned int lo, hi;
__asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));
return ((uint64_t)hi << 32) | lo;
}
#else
static inline uint64_t tiny_fast_rdtsc(void) { return 0; }
#endif
extern int g_profile_enabled;
static inline int tiny_fast_profile_enabled(void) {
extern int g_profile_enabled;
if (__builtin_expect(g_profile_enabled == -1, 0)) {
const char* env = getenv("HAKMEM_TINY_PROFILE");
g_profile_enabled = (env && *env && *env != '0') ? 1 : 0;
}
return g_profile_enabled;
}
// ========== Size to Class Mapping ========== // ========== Size to Class Mapping ==========
// Inline size-to-class for fast path (minimal branches) // Inline size-to-class for fast path (O(1) lookup table)
static inline int tiny_fast_size_to_class(size_t size) { static inline int tiny_fast_size_to_class(size_t size) {
// Class mapping (same as existing Tiny classes): // Optimized: Lookup table for O(1) mapping (vs 11-branch linear search)
// 0: 16B, 1: 24B, 2: 32B, 3: 40B, 4: 48B, 5: 56B, 6: 64B // Table indexed by (size >> 3) for sizes 0-128
// 7: 80B, 8: 96B, 9: 112B, 10: 128B, 11-15: reserved // Class mapping: 0:16B, 1:24B, 2:32B, 3:40B, 4:48B, 5:56B, 6:64B, 7:80B, 8:96B, 9:112B, 10:128B
if (size <= 16) return 0; static const int8_t size_to_class_lut[17] = {
if (size <= 24) return 1; 0, // 0-7 → 16B (class 0)
if (size <= 32) return 2; 0, // 8-15 → 16B (class 0)
if (size <= 40) return 3; 0, // 16 → 16B (class 0)
if (size <= 48) return 4; 1, // 17-23 → 24B (class 1)
if (size <= 56) return 5; 1, // 24 → 24B (class 1)
if (size <= 64) return 6; 2, // 25-31 → 32B (class 2)
if (size <= 80) return 7; 2, // 32 → 32B (class 2)
if (size <= 96) return 8; 3, // 33-39 → 40B (class 3)
if (size <= 112) return 9; 3, // 40 → 40B (class 3)
if (size <= 128) return 10; 4, // 41-47 → 48B (class 4)
return -1; // Not tiny 4, // 48 → 48B (class 4)
5, // 49-55 → 56B (class 5)
5, // 56 → 56B (class 5)
6, // 57-63 → 64B (class 6)
6, // 64 → 64B (class 6)
7, // 65-79 → 80B (class 7)
8 // 80-95 → 96B (class 8)
};
if (__builtin_expect(size > 128, 0)) return -1; // Not tiny
// Fast path: Direct lookup (1-2 instructions!)
unsigned int idx = size >> 3; // size / 8
if (__builtin_expect(idx < 17, 1)) {
return size_to_class_lut[idx];
}
// Size 96-128: class 9-10
if (size <= 112) return 9; // 112B (class 9)
return 10; // 128B (class 10)
} }
// ========== Forward Declarations ========== // ========== Forward Declarations ==========
@ -66,40 +123,97 @@ void tiny_fast_drain(int class_idx);
// ========== Fast Path: Alloc (3-4 instructions!) ========== // ========== Fast Path: Alloc (3-4 instructions!) ==========
static inline void* tiny_fast_alloc(size_t size) { static inline void* tiny_fast_alloc(size_t size) {
uint64_t start = tiny_fast_profile_enabled() ? tiny_fast_rdtsc() : 0;
// Step 1: Size to class (1-2 instructions, branch predictor friendly) // Step 1: Size to class (1-2 instructions, branch predictor friendly)
int cls = tiny_fast_size_to_class(size); int cls = tiny_fast_size_to_class(size);
if (__builtin_expect(cls < 0, 0)) return NULL; // Not tiny (rare) if (__builtin_expect(cls < 0, 0)) return NULL; // Not tiny (rare)
// Step 2: Pop from TLS cache (2-3 instructions) // Step 2: Pop from alloc_head (hot allocation path)
void* ptr = g_tiny_fast_cache[cls]; void* ptr = g_tiny_fast_cache[cls];
if (__builtin_expect(ptr != NULL, 1)) { if (__builtin_expect(ptr != NULL, 1)) {
// Fast path: Pop head, decrement count // Fast path: Pop head, decrement count
g_tiny_fast_cache[cls] = *(void**)ptr; g_tiny_fast_cache[cls] = *(void**)ptr;
g_tiny_fast_count[cls]--; g_tiny_fast_count[cls]--;
if (start) {
g_tiny_malloc_cycles += (tiny_fast_rdtsc() - start);
g_tiny_malloc_count++;
}
return ptr;
}
// ========================================================================
// Phase 6-7: Step 2.5: Lazy Migration from free_head (Phase 2)
// If alloc_head empty but free_head has blocks, migrate with pointer swap
// This is mimalloc's key optimization: batched migration, zero overhead
// ========================================================================
if (__builtin_expect(g_tiny_fast_free_head[cls] != NULL, 0)) {
uint64_t mig_start = start ? tiny_fast_rdtsc() : 0;
// Migrate entire free_head → alloc_head (pointer swap, instant!)
g_tiny_fast_cache[cls] = g_tiny_fast_free_head[cls];
g_tiny_fast_count[cls] = g_tiny_fast_free_count[cls];
g_tiny_fast_free_head[cls] = NULL;
g_tiny_fast_free_count[cls] = 0;
// Now pop one from newly migrated list
ptr = g_tiny_fast_cache[cls];
g_tiny_fast_cache[cls] = *(void**)ptr;
g_tiny_fast_count[cls]--;
if (mig_start) {
g_tiny_migration_cycles += (tiny_fast_rdtsc() - mig_start);
g_tiny_migration_count++;
}
if (start) {
g_tiny_malloc_cycles += (tiny_fast_rdtsc() - start);
g_tiny_malloc_count++;
}
return ptr; return ptr;
} }
// Step 3: Slow path - refill from Magazine/SuperSlab // Step 3: Slow path - refill from Magazine/SuperSlab
return tiny_fast_refill(cls); ptr = tiny_fast_refill(cls);
if (start) {
g_tiny_malloc_cycles += (tiny_fast_rdtsc() - start);
g_tiny_malloc_count++;
}
return ptr;
} }
// ========== Fast Path: Free (2-3 instructions!) ========== // ========== Fast Path: Free (2-3 instructions!) ==========
static inline void tiny_fast_free(void* ptr, size_t size) { static inline void tiny_fast_free(void* ptr, size_t size) {
uint64_t start = tiny_fast_profile_enabled() ? tiny_fast_rdtsc() : 0;
// Step 1: Size to class // Step 1: Size to class
int cls = tiny_fast_size_to_class(size); int cls = tiny_fast_size_to_class(size);
if (__builtin_expect(cls < 0, 0)) return; // Not tiny (error) if (__builtin_expect(cls < 0, 0)) return; // Not tiny (error)
// Step 2: Check capacity // ========================================================================
if (__builtin_expect(g_tiny_fast_count[cls] >= TINY_FAST_CACHE_CAP, 0)) { // Phase 6-7: Push to free_head (Phase 2)
// Cache full - drain to Magazine/SuperSlab // Separate free staging area reduces cache line contention with alloc_head
// mimalloc's key insight: alloc/free touch different cache lines
// ========================================================================
// Step 2: Check free_head capacity
if (__builtin_expect(g_tiny_fast_free_count[cls] >= TINY_FAST_CACHE_CAP, 0)) {
// Free cache full - drain to Magazine/SuperSlab
tiny_fast_drain(cls); tiny_fast_drain(cls);
} }
// Step 3: Push to TLS cache (2 instructions) // Step 3: Push to free_head (separate cache line from alloc_head!)
*(void**)ptr = g_tiny_fast_cache[cls]; *(void**)ptr = g_tiny_fast_free_head[cls];
g_tiny_fast_cache[cls] = ptr; g_tiny_fast_free_head[cls] = ptr;
g_tiny_fast_count[cls]++; g_tiny_fast_free_count[cls]++;
if (start) {
g_tiny_free_cycles += (tiny_fast_rdtsc() - start);
g_tiny_free_count++;
}
} }
// ========== Initialization ========== // ========== Initialization ==========
@ -109,5 +223,10 @@ static inline void tiny_fast_init(void) {
memset(g_tiny_fast_cache, 0, sizeof(g_tiny_fast_cache)); memset(g_tiny_fast_cache, 0, sizeof(g_tiny_fast_cache));
memset(g_tiny_fast_count, 0, sizeof(g_tiny_fast_count)); memset(g_tiny_fast_count, 0, sizeof(g_tiny_fast_count));
// Phase 6-7: Initialize dual free lists (Phase 2)
memset(g_tiny_fast_free_head, 0, sizeof(g_tiny_fast_free_head));
memset(g_tiny_fast_free_count, 0, sizeof(g_tiny_fast_free_count));
g_tiny_fast_initialized = 1; g_tiny_fast_initialized = 1;
} }

1
mimalloc-bench Submodule

Submodule mimalloc-bench added at 6ec12891f8

25
scripts/profiles/tinyhot_optimized.env Normal file
View File

@ -0,0 +1,25 @@
# CLAUDE.md optimized settings for Larson
export HAKMEM_TINY_FAST_PATH=1
export HAKMEM_TINY_USE_SUPERSLAB=1
export HAKMEM_USE_SUPERSLAB=1
export HAKMEM_TINY_SS_ADOPT=1
export HAKMEM_WRAP_TINY=1
# Key optimizations from CLAUDE.md
export HAKMEM_TINY_FAST_CAP=16 # Reduced from 64
export HAKMEM_TINY_FAST_CAP_0=16
export HAKMEM_TINY_FAST_CAP_1=16
export HAKMEM_TINY_REFILL_COUNT_HOT=64
# Disable magazine layers
export HAKMEM_TINY_TLS_SLL=1
export HAKMEM_TINY_TLS_LIST=0
export HAKMEM_TINY_HOTMAG=0
# Debug OFF
export HAKMEM_TINY_TRACE_RING=0
export HAKMEM_SAFE_FREE=0
export HAKMEM_TINY_REMOTE_GUARD=0
export HAKMEM_DEBUG_COUNTERS=0
export HAKMEM_TINY_PHASE6_BOX_REFACTOR=1