hakmem

tomoaki/hakmem

Fork 0

Commit Graph

Author	SHA1	Message	Date
Moe Charm (CI)	707056b765	feat: Phase 7 + Phase 2 - Massive performance & stability improvements Performance Achievements: - Tiny allocations: +180-280% (21M → 59-70M ops/s random mixed) - Single-thread: +24% (2.71M → 3.36M ops/s Larson) - 4T stability: 0% → 95% (19/20 success rate) - Overall: 91.3% of System malloc average (target was 40-55%) ✓ Phase 7 (Tasks 1-3): Core Optimizations - Task 1: Header validation removal (Region-ID direct lookup) - Task 2: Aggressive inline (TLS cache access optimization) - Task 3: Pre-warm TLS cache (eliminate cold-start penalty) Result: +180-280% improvement, 85-146% of System malloc Critical Bug Fixes: - Fix 64B allocation crash (size-to-class +1 for header) - Fix 4T wrapper recursion bugs (BUG #7, #8, #10, #11) - Remove malloc fallback (30% → 50% stability) Phase 2a: SuperSlab Dynamic Expansion (CRITICAL) - Implement mimalloc-style chunk linking - Unlimited slab expansion (no more OOM at 32 slabs) - Fix chunk initialization bug (bitmap=0x00000001 after expansion) Files: core/hakmem_tiny_superslab.c/h, core/superslab/superslab_types.h Result: 50% → 95% stability (19/20 4T success) Phase 2b: TLS Cache Adaptive Sizing - Dynamic capacity: 16-2048 slots based on usage - High-water mark tracking + exponential growth/shrink - Expected: +3-10% performance, -30-50% memory Files: core/tiny_adaptive_sizing.c/h (new) Phase 2c: BigCache Dynamic Hash Table - Migrate from fixed 256×8 array to dynamic hash table - Auto-resize: 256 → 512 → 1024 → 65,536 buckets - Improved hash function (FNV-1a) + collision chaining Files: core/hakmem_bigcache.c/h Expected: +10-20% cache hit rate Design Flaws Analysis: - Identified 6 components with fixed-capacity bottlenecks - SuperSlab (CRITICAL), TLS Cache (HIGH), BigCache/L2.5 (MEDIUM) - Report: DESIGN_FLAWS_ANALYSIS.md (11 chapters) Documentation: - 13 comprehensive reports (PHASE.md, DESIGN_FLAWS.md) - Implementation guides, test results, production readiness - Bug fix reports, root cause analysis Build System: - Makefile: phase7 targets, PREWARM_TLS flag - Auto dependency generation (-MMD -MP) for .inc files Known Issues: - 4T stability: 19/20 (95%) - investigating 1 failure for 100% - L2.5 Pool dynamic sharding: design only (needs 2-3 days integration) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-08 17:08:00 +09:00
Moe Charm (CI)	f454d35ea4	Perf: getenv ホットパスボトルネック削除 (8.51% → 0%) 問題: perf で発見: - `getenv()`: 8.51% CPU on malloc hot path - malloc 内で `getenv("HAKMEM_SFC_DEBUG")` が毎回実行 - getenv は環境変数の線形走査 → 非常に重い修正内容: 1. `malloc()`: HAKMEM_SFC_DEBUG を初回のみ getenv して cache (Line 48-52) 2. `malloc()`: HAKMEM_LD_SAFE を初回のみ getenv して cache (Line 75-79) 3. `calloc()`: HAKMEM_LD_SAFE を初回のみ getenv して cache (Line 120-124) 効果: - getenv CPU: 8.51% → 0% ✅ - superslab_refill: 10.30% → 9.61% (-7%) - hak_tiny_alloc_slow が新トップ: 9.61% スループット: - 4,192,132 ops/s (変化なし) - 理由: Syscall Saturation (86.7% kernel time) が支配的 - 次: SuperSlab Caching で syscall 90% 削減 → +100-150% 期待 Perf結果 (before/after): ``` Before: getenv 8.51% \| superslab_refill 10.30% After: getenv 0% \| hak_tiny_alloc_slow 9.61% \| superslab_refill 9.61% ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-07 01:15:28 +09:00
Moe Charm (CI)	db833142f1	Fix: malloc 初期化デッドロックを解消問題: - Larson ベンチマークが起動時に futex でハング - 全プロセスが FUTEX_WAIT_PRIVATE で永遠に待機 - 初期化が完了せず、何も出力されない根本原因: `core/box/hak_wrappers.inc.h` の `malloc()` 関数で、 Line 42 の `getenv("HAKMEM_SFC_DEBUG")` が `g_initializing` チェックより前に実行される → `getenv()` が内部で malloc を呼ぶ → 無限再帰 → pthread_once デッドロック修正内容: `g_initializing` チェックを malloc() の最初に移動 (Line 41-44) - 初期化中の再帰呼び出しを即座に libc にフォールバック - getenv() などの init 関数が malloc を呼んでも安全効果: - デッドロック完全解消 ✅ - Larson ベンチマーク正常起動 - 性能維持: 4,192,124 ops/s (4.19M baseline) テスト: ```bash ./larson_hakmem 1 8 128 128 1 1 1 # → 367,082 ops/s ✅ ./larson_hakmem 2 8 128 1024 1 12345 4 # → 4,192,124 ops/s ✅ ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-07 00:37:33 +09:00

Author

SHA1

Message

Date

Moe Charm (CI)

707056b765

feat: Phase 7 + Phase 2 - Massive performance & stability improvements

Performance Achievements:
- Tiny allocations: +180-280% (21M → 59-70M ops/s random mixed)
- Single-thread: +24% (2.71M → 3.36M ops/s Larson)
- 4T stability: 0% → 95% (19/20 success rate)
- Overall: 91.3% of System malloc average (target was 40-55%) ✓

Phase 7 (Tasks 1-3): Core Optimizations
- Task 1: Header validation removal (Region-ID direct lookup)
- Task 2: Aggressive inline (TLS cache access optimization)
- Task 3: Pre-warm TLS cache (eliminate cold-start penalty)
  Result: +180-280% improvement, 85-146% of System malloc

Critical Bug Fixes:
- Fix 64B allocation crash (size-to-class +1 for header)
- Fix 4T wrapper recursion bugs (BUG #7, #8, #10, #11)
- Remove malloc fallback (30% → 50% stability)

Phase 2a: SuperSlab Dynamic Expansion (CRITICAL)
- Implement mimalloc-style chunk linking
- Unlimited slab expansion (no more OOM at 32 slabs)
- Fix chunk initialization bug (bitmap=0x00000001 after expansion)
  Files: core/hakmem_tiny_superslab.c/h, core/superslab/superslab_types.h
  Result: 50% → 95% stability (19/20 4T success)

Phase 2b: TLS Cache Adaptive Sizing
- Dynamic capacity: 16-2048 slots based on usage
- High-water mark tracking + exponential growth/shrink
- Expected: +3-10% performance, -30-50% memory
  Files: core/tiny_adaptive_sizing.c/h (new)

Phase 2c: BigCache Dynamic Hash Table
- Migrate from fixed 256×8 array to dynamic hash table
- Auto-resize: 256 → 512 → 1024 → 65,536 buckets
- Improved hash function (FNV-1a) + collision chaining
  Files: core/hakmem_bigcache.c/h
  Expected: +10-20% cache hit rate

Design Flaws Analysis:
- Identified 6 components with fixed-capacity bottlenecks
- SuperSlab (CRITICAL), TLS Cache (HIGH), BigCache/L2.5 (MEDIUM)
- Report: DESIGN_FLAWS_ANALYSIS.md (11 chapters)

Documentation:
- 13 comprehensive reports (PHASE*.md, DESIGN_FLAWS*.md)
- Implementation guides, test results, production readiness
- Bug fix reports, root cause analysis

Build System:
- Makefile: phase7 targets, PREWARM_TLS flag
- Auto dependency generation (-MMD -MP) for .inc files

Known Issues:
- 4T stability: 19/20 (95%) - investigating 1 failure for 100%
- L2.5 Pool dynamic sharding: design only (needs 2-3 days integration)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-08 17:08:00 +09:00

Moe Charm (CI)

f454d35ea4

Perf: getenv ホットパスボトルネック削除 (8.51% → 0%)

**問題:**
perf で発見:
- `getenv()`: 8.51% CPU on malloc hot path
- malloc 内で `getenv("HAKMEM_SFC_DEBUG")` が毎回実行
- getenv は環境変数の線形走査 → 非常に重い

**修正内容:**
1. `malloc()`: HAKMEM_SFC_DEBUG を初回のみ getenv して cache (Line 48-52)
2. `malloc()`: HAKMEM_LD_SAFE を初回のみ getenv して cache (Line 75-79)
3. `calloc()`: HAKMEM_LD_SAFE を初回のみ getenv して cache (Line 120-124)

**効果:**
- getenv CPU: 8.51% → 0% ✅
- superslab_refill: 10.30% → 9.61% (-7%)
- hak_tiny_alloc_slow が新トップ: 9.61%

**スループット:**
- 4,192,132 ops/s (変化なし)
- 理由: Syscall Saturation (86.7% kernel time) が支配的
- 次: SuperSlab Caching で syscall 90% 削減 → +100-150% 期待

**Perf結果 (before/after):**
```
Before:  getenv 8.51% | superslab_refill 10.30%
After:   getenv 0%    | hak_tiny_alloc_slow 9.61% | superslab_refill 9.61%
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-07 01:15:28 +09:00

Moe Charm (CI)

db833142f1

Fix: malloc 初期化デッドロックを解消

**問題:**
- Larson ベンチマークが起動時に futex でハング
- 全プロセスが FUTEX_WAIT_PRIVATE で永遠に待機
- 初期化が完了せず、何も出力されない

**根本原因:**
`core/box/hak_wrappers.inc.h` の `malloc()` 関数で、
Line 42 の `getenv("HAKMEM_SFC_DEBUG")` が `g_initializing` チェックより前に実行される
→ `getenv()` が内部で malloc を呼ぶ
→ 無限再帰 → pthread_once デッドロック

**修正内容:**
`g_initializing` チェックを malloc() の最初に移動 (Line 41-44)
- 初期化中の再帰呼び出しを即座に libc にフォールバック
- getenv() などの init 関数が malloc を呼んでも安全

**効果:**
- デッドロック完全解消 ✅
- Larson ベンチマーク正常起動
- 性能維持: 4,192,124 ops/s (4.19M baseline)

**テスト:**
```bash
./larson_hakmem 1 8 128 128 1 1 1        # → 367,082 ops/s ✅
./larson_hakmem 2 8 128 1024 1 12345 4  # → 4,192,124 ops/s ✅
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-07 00:37:33 +09:00

3 Commits