tomoaki/hakmem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
465 Commits 2 Branches 0 Tags
9c24bebf08a7207933d13e2d3fb23bfc890f975d
Commit Graph

370 Commits

Author SHA1 Message Date
Moe Charm (CI)
9c24bebf08 Phase v5-1: SmallObject v5 C6-only route stub 接続 
- tiny_route_env_box.h: TINY_ROUTE_SMALL_HEAP_V5 enum 追加、route snapshot で C6→v5 分岐
- malloc_tiny_fast.h: alloc/free switch に v5 case 追加（v1/pool fallback）
- smallobject_hotbox_v5.c: stub 実装（alloc は NULL 返却、free は no-op）
- smallobject_hotbox_v5_box.h: 関数 signature に ctx パラメータ追加
- Makefile: core/smallobject_hotbox_v5.o をリンクリストに追加
- ENV_PROFILE_PRESETS.md: v5-1 プリセット追記
- CURRENT_TASK.md: Phase v5-1 完了記録

**特性**:
- ENV: HAKMEM_SMALL_HEAP_V5_ENABLED=1 / HAKMEM_SMALL_HEAP_V5_CLASSES=0x40 で opt-in
- テスト結果: C6-heavy (v5 OFF 15.5M → v5 ON 16.4M ops/s, 正常), Mixed 47.2M ops/s, SEGV/assert なし
- 挙動は v1/pool fallback と同じ（実装は v5-2）

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-11 03:25:37 +09:00
Moe Charm (CI)
dedfea27d5 Phase v5-0 refactor: ENV統一・マクロ化・構造体最適化 
- ENV initialization を sentinel パターンで統一
  - ENV_UNINIT/ENABLED/DISABLED 定数追加
  - __builtin_expect で初期化チェックを最適化
  - small_heap_v5_enabled/class_mask を統一パターンに変更

- ポインタマクロ化（O(1) segment/page 計算）
  - SMALL_SEGMENT_V5_BASE_FROM_PTR: ptr から segment base を mask で計算
  - SMALL_SEGMENT_V5_PAGE_IDX: segment 内の page_idx を shift で計算
  - SMALL_SEGMENT_V5_PAGE_META: page_meta への O(1) access（bounds check付き）
  - SMALL_SEGMENT_V5_VALIDATE_MAGIC: magic 検証
  - SMALL_SEGMENT_V5_VALIDATE_PTR: Fail-Fast validation pipeline

- SmallClassHeapV5 に partial_count 追加
  - partial ページリストのカウンタを追加（refill/retire 最適化用）

- SmallPageMetaV5 の field 再配置（L1 cache 最適化）
  - hot fields (free_list, used, capacity) を先頭に集約
  - metadata (class_idx, flags, page_idx, segment) を後方配置
  - total 24B、offset コメント追加

- route priority ENV 追加
  - HAKMEM_ROUTE_PRIORITY={v4|v5|auto}（default: v4）
  - enum small_route_priority 定義
  - small_route_priority() 関数追加

- segment_size override ENV 追加
  - HAKMEM_SMALL_HEAP_V5_SEGMENT_SIZE（default: 2MiB）
  - power of 2 & >= 64KiB validation

挙動: 完全不変（v5 route は呼ばれない、ENV default OFF）
テスト: Mixed 16–1024B で 43.0–43.8M ops/s（変化なし）、SEGV/assert なし

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-11 03:19:18 +09:00
Moe Charm (CI)
83d4096fbc Phase v5-0: SmallObject v5 の設計・型/IF/ENV スケルトン追加 
設計ドキュメント:
- docs/analysis/SMALLOBJECT_V5_DESIGN.md: v5 アーキテクチャ全体設計

新規ファイル (v5 スケルトン):
- core/box/smallobject_hotbox_v5_box.h: HotBox v5 型定義
- core/box/smallsegment_v5_box.h: Segment v5 型定義
- core/box/smallobject_cold_iface_v5.h: ColdIface v5 IF宣言
- core/box/smallobject_v5_env_box.h: ENV ゲート
- core/smallobject_hotbox_v5.c: 実装 stub (完全 fallback)

特徴:
✅ 型とインターフェースのみ定義（v5-0 は機能なし）
✅ ENV デフォルト OFF（HAKMEM_SMALL_HEAP_V5_ENABLED=0）
✅ 挙動完全不変（Mixed/C6 benchmark 確認済み）
✅ v4 との区別を明確化 (*_v5 suffix)
✅ v5-1 (stub) → v5-2 (本実装) → v5-3 (Mixed) への段階実装準備完了

フェーズ:
- v5-0: 型定義のみ（現在）
- v5-1: C6-only stub route 追加
- v5-2: Segment/HotBox 本実装 (C6-only bench A/B)
- v5-3: Mixed での段階昇格 (C6 → C5 → ...)

目標性能: Mixed 16–1024B で 50–60M ops/s (mimalloc の 5割)

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-11 03:09:57 +09:00
Moe Charm (CI)
bdfa32d869 Phase v4-mid-SEGV: C6 v4 を SmallSegment 専用に切り替え、TinyHeap SEGV を解決 
問題: C6 v4 が TinyHeap のページを共有することで iters >= 800k で freelist 破壊 → SEGV 発生

修正内容:
- c6_segment_alloc_page_direct(): C6 専用ページ割当 (SmallSegment v4 経由, TinyHeap 非共有)
- c6_segment_release_page_direct(): C6 専用ページ返却
- cold_refill_page_v4() で C6 を分岐: SmallSegment 直接使用
- cold_retire_page_v4() で C6 を分岐: SmallSegment に直接返却
- fastlist state reset 処理追加 (L392-399)

結果:
✅ iters=1M, ws <= 390 で SEGV 消失
✅ C6-only: v4 OFF ~47M → v4 ON ~43M ops/s (−8.5%, 安定)
✅ Mixed: v4 ON で SEGV なし (小幅回帰許容)

方針: C6 v4 は研究箱として安定化完了。本線には載せない (既存 mid/pool v1 使用)。

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-11 02:39:32 +09:00
Moe Charm (CI)
e486dd2c55 Phase v4-mid-6: Implement C6 v4 TLS Fastlist (Gated) 
- Implemented TLS fastlist logic for C6 in smallobject_hotbox_v4.c (alloc/free).
- Added SmallC6FastState struct and g_small_c6_fast TLS variable.
- Gated the fastlist logic with HAKMEM_SMALL_HEAP_V4_FASTLIST (default OFF) due to observed instability in mixed workloads.
- Fixed a memory leak in small_heap_free_fast_v4 fallback path by calling hak_pool_free.
- Updated CURRENT_TASK.md with phase report.
 2025-12-11 01:44:08 +09:00
Moe Charm (CI)
dd974b49c5 Phase v4-mid-2, v4-mid-3, v4-mid-5: SmallObject HotBox v4 implementation and docs update 
Implementation:
- SmallObject HotBox v4 (core/smallobject_hotbox_v4.c) now fully implements C6-only allocations and frees, including current/partial management and freelist operations.
- Cold Iface (tiny_heap based) for page refill/retire is integrated.
- Stats instrumentation (v4-mid-5) added to small_heap_alloc_fast_v4 and small_heap_free_fast_v4, with a new header file core/box/smallobject_hotbox_v4_stats_box.h and atexit dump function.

Updates:
- CURRENT_TASK.md has been condensed and updated with summaries of Phase v4-mid-2 (C6-only v4), Phase v4-mid-3 (C5-only v4 pilot), and the stats implementation (v4-mid-5).
- docs/analysis/SMALLOBJECT_V4_BOX_DESIGN.md updated with A/B results and conclusions for C6-only and C5-only v4 implementations.
- The previous CURRENT_TASK.md content has been archived to CURRENT_TASK_ARCHIVE_20251210.md.
 2025-12-11 01:01:15 +09:00
Moe Charm (CI)
3b4449d773 Phase v4-mid-1: C6-only v4 route + page_meta_of() Fail-Fast validation 
Implementation:
- SMALL_SEGMENT_V4_* constants (SIZE=2MiB, PAGE_SIZE=64KiB, MAGIC=0xDEADBEEF)
- smallsegment_v4_page_meta_of(): O(1) mask+shift lookup with magic validation
  - Computes segment base: addr & ~(2MiB - 1)
  - Verifies SmallSegment magic number
  - Calculates page_idx: (addr - seg_base) >> PAGE_SHIFT (16)
  - Returns non-NULL sentinel for now (full page_meta[] in Phase v4-mid-2)

Stubs for C6-only phase:
- small_heap_alloc_fast_v4(): C6 returns NULL → pool v1 fallback
- small_heap_free_fast_v4(): C6 calls page_meta_of() for Fail-Fast, then pool v1 fallback

Documentation:
- ENV_PROFILE_PRESETS.md: Add "C6_ONLY_SMALLOBJECT_V4" research profile
  - HAKMEM_SMALL_HEAP_V4_ENABLED=1, HAKMEM_SMALL_HEAP_V4_CLASSES=0x40
  - Expected: Throughput ≈ 28–29M ops/s (same as v1)

Build:
- ビルド成功（警告のみ）
- Backward compatible, alloc/free stubs fall back to pool v1

Sanity:
- C6-heavy with v4 opt-in: segv/assert なし
- page_meta_of() lookup working correctly
- Performance unchanged (expected for stub phase)

Status:
- C6-only v4 route now available via ENV opt-in
- Phase v4-mid-2: SmallHeapCtx v4 full implementation with A/B

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 23:37:45 +09:00
Moe Charm (CI)
e3e4cab833 Cleanup: Unify type naming and Cold Iface architecture 
Refactoring:
- Type naming: Rename small_page_v4 → SmallPageMeta, small_class_heap_v4 → SmallClassHeap, small_heap_ctx_v4 → SmallHeapCtx
- Keep backward compatibility aliases for existing code
- SmallSegment struct unified, clean forward declarations
- Cold Iface: Remove vtable (SmallColdIfaceV4 struct) in favor of direct function calls
- Simplify refill_page/retire_page to direct calls, not callbacks
- smallobject_hotbox_v4.c: Update to call small_cold_v4_* functions directly

Documentation:
- Add docs/analysis/ENV_CLEANUP_CANDIDATES.md
- Categorize ENVs: KEEP (production), RESEARCH (opt-in), DELETE (obsolete)
- v2 code: Keep as research infrastructure (complete, safe, gated)
- v4 code: Research scaffold for future mid-level allocator

Build:
- ビルド成功（警告のみ）
- Backward compatible, all existing code still works

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 23:30:32 +09:00
Moe Charm (CI)
52c65da783 Phase v4-mid-0: Small-object v4 型・IF 足場（箱化モジュール化） 
- SmallHeapCtx/SmallPageMeta/SmallClassHeap typedef alias 追加
- SmallSegment struct (base/num_pages/owner_tid/magic) を smallsegment_v4_box.h に定義
- SmallColdIface_v4 direct function prototypes (refill/retire/remote_push/drain)
- smallobject_hotbox_v4.c の internal/public API 分離（small_segment_v4_internal）
- direct function stubs 実装（SmallColdIfaceV4 delegate 形式）
- ENV OFF デフォルト（ENABLED=0/CLASSES=0）で既存挙動 100% 不変
- ビルド成功・sanity 確認（mixed/C6-heavy、segv/assert なし）
- CURRENT_TASK.md に Phase v4-mid-0 記録

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 23:23:07 +09:00
Moe Charm (CI)
2a13478dc7 Optimize C6 heavy and C7 ultra performance analysis with refined design refinements 
- Update environment profile presets and visibility analysis
- Enhance small object and tiny segment v4 box implementations
- Refine C7 ultra and C6 heavy allocation strategies
- Add comprehensive performance metrics and design documentation

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 22:57:26 +09:00
Moe Charm (CI)
9460785bd6 Enable C7 ULTRA segment path by default 2025-12-10 22:25:24 +09:00
Moe Charm (CI)
bbb55b018a Add C7 ULTRA segment skeleton and TLS freelist 2025-12-10 22:19:32 +09:00
Moe Charm (CI)
f2ce7256cd Add v4 C7/C6 fast classify and small-segment v4 scaffolding 2025-12-10 19:14:38 +09:00
Moe Charm (CI)
3261025995 Phase v4-4: pilot C6 v4 route with opt-in gate 2025-12-10 18:18:05 +09:00
Moe Charm (CI)
7be30c0b5a Avoid full-list scans for C7 v4 and tighten partial reuse 2025-12-10 18:04:32 +09:00
Moe Charm (CI)
860d934d71 Tune C7 v4 partial reuse for mixed perf 2025-12-10 18:03:28 +09:00
Moe Charm (CI)
cbd33511eb Phase v4-3.1: reuse C7 v4 pages and record prep calls 2025-12-10 17:58:42 +09:00
Moe Charm (CI)
406a2f4d26 Incremental improvements: mid_desc cache, pool hotpath optimization, and doc updates 
**Changes:**
- core/box/pool_api.inc.h: Code organization and micro-optimizations
- CURRENT_TASK.md: Updated Phase MD1 (mid_desc TLS cache: +3.2% for C6-heavy)
- docs/analysis files: Various analysis and documentation updates
- AGENTS.md: Agent role clarifications
- TINY_FRONT_V3_FLATTENING_GUIDE.md: Flattening strategy documentation

**Verification:**
- random_mixed_hakmem: 44.8M ops/s (1M iterations, 400 working set)
- No segfaults or assertions across all benchmark variants
- Stable performance across multiple runs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 14:00:57 +09:00
Moe Charm (CI)
0e5a2634bc Phase 82 Final: Documentation of mid_desc race fix and comprehensive A/B results 
**Implementation Summary:**
- Early `mid_desc_init_once()` in `hak_pool_init_impl()` prevents uninitialized mutex crash
- Eliminates race condition that caused C7_SAFE + flatten crashes
- Enables safe operation across all profiles (C7_SAFE, LEGACY)

**Benchmark Results (C6_HEAVY_LEGACY_POOLV1, Release):**
- Phase 1 (Baseline): 3.03M / 14.86M / 26.67M ops/s (10K/100K/1M)
- Phase 2 (Zero Mode): +5.0% / -2.7% / -0.2%
- Phase 3 (Flatten): +3.7% / +6.1% / -5.0%
- Phase 4 (Combined): -5.1% / +8.8% / +2.0% (best at 100K: +8.8%)
- Phase 5 (C7_SAFE Safety): NO CRASH ✅ (all iterations stable)

**Mainline Policy:**
- mid_desc initialization: Always enabled (crash prevention)
- Flatten: Default OFF (bench opt-in via HAKMEM_POOL_V1_FLATTEN_ENABLED=1)
- Zero Mode: Default FULL (bench opt-in via HAKMEM_POOL_ZERO_MODE=header)
- Workload-specific: Medium (100K) benefits most (+8.8%)

**Documentation Updated:**
- CURRENT_TASK.md: Added Phase 82 conclusions with benchmark table
- MID_LARGE_CPU_HOTPATH_ANALYSIS.md: Added Phase 82 Final with workload analysis

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 09:35:18 +09:00
Moe Charm (CI)
ae056e26ae Phase ML1 refactoring: Code readability and warnings cleanup 
- Add (void) casts for unused timespec/profiling variables
- Split multi-statement lines in pool_free_fast functions for clarity
- Mark pool_hotbox_v2_pop_partial as __attribute__((unused))
- Verified functionality with HAKMEM_POOL_ZERO_MODE=header optimization
- Performance stable: +16.1% improvement in header mode (10K iterations)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 09:15:24 +09:00
Moe Charm (CI)
acc64f2438 Phase ML1: Pool v1 memset 89.73% overhead 軽量化 (+15.34% improvement) 
## Summary
- ChatGPT により bench_profile.h の setenv segfault を修正（RTLD_NEXT 経由に切り替え）
- core/box/pool_zero_mode_box.h 新設：ENV キャッシュ経由で ZERO_MODE を統一管理
- core/hakmem_pool.c で zero mode に応じた memset 制御（FULL/header/off）
- A/B テスト結果：ZERO_MODE=header で +15.34% improvement（1M iterations, C6-heavy）

## Files Modified
- core/box/pool_api.inc.h: pool_zero_mode_box.h include
- core/bench_profile.h: glibc setenv → malloc+putenv（segfault 回避）
- core/hakmem_pool.c: zero mode 参照・制御ロジック
- core/box/pool_zero_mode_box.h (新設): enum/getter
- CURRENT_TASK.md: Phase ML1 結果記載

## Test Results
| Iterations | ZERO_MODE=full | ZERO_MODE=header | Improvement |
|-----------|----------------|-----------------|------------|
| 10K       | 3.06 M ops/s   | 3.17 M ops/s    | +3.65%     |
| 1M        | 23.71 M ops/s  | 27.34 M ops/s   | **+15.34%** |

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2025-12-10 09:08:18 +09:00
Moe Charm (CI)
a905e0ffdd Guard madvise ENOMEM and stabilize pool/tiny front v3 2025-12-09 21:50:15 +09:00
Moe Charm (CI)
e274d5f6a9 pool v1 flatten: break down free fallback causes and normalize mid_desc keys 2025-12-09 19:34:54 +09:00
Moe Charm (CI)
8f18963ad5 Phase 36-37: TinyHotHeap v2 HotBox redesign and C7 current_page policy fixes 
- Redefine TinyHotHeap v2 as per-thread Hot Box with clear boundaries
- Add comprehensive OS statistics tracking for SS allocations
- Implement route-based free handling for TinyHeap v2
- Add C6/C7 debugging and statistics improvements
- Update documentation with implementation guidelines and analysis
- Add new box headers for stats, routing, and front-end management
 2025-12-08 21:30:21 +09:00
Moe Charm (CI)
34a8fd69b6 C7 v2: add lease helpers and v2 page reset 2025-12-08 14:40:03 +09:00
Moe Charm (CI)
9502501842 Fix tiny lane success handling for TinyHeap routes 2025-12-07 23:06:50 +09:00
Moe Charm (CI)
a6991ec9e4 Add TinyHeap class mask and extend routing 2025-12-07 22:49:28 +09:00
Moe Charm (CI)
9c68073557 C7 meta-light delta flush threshold and clamp 2025-12-07 22:42:02 +09:00
Moe Charm (CI)
fda6cd2e67 Boxify superslab registry, add bench profile, and document C7 hotpath experiments 2025-12-07 03:12:27 +09:00
Moe Charm (CI)
18faa6a1c4 Add OBSERVE stats and auto tiny policy profile 2025-12-06 01:44:05 +09:00
Moe Charm (CI)
03538055ae Restore C7 Warm/TLS carve for release and add policy scaffolding 2025-12-06 01:34:04 +09:00
Moe Charm (CI)
d17ec46628 Fix C7 warm/TLS Release path and unify debug instrumentation 2025-12-05 23:41:01 +09:00
Moe Charm (CI)
e96e9a4bf9 Feat: Add TLS carve experiment for warm C7 2025-12-05 20:50:24 +09:00
Moe Charm (CI)
3e1d7c3798 Fix debug build after clean reset 2025-12-05 20:43:14 +09:00
Moe Charm (CI)
4c986fa9d1 Feat: Add experimental TLS Bind Box path in Unified Cache 
- Added experimental path in unified_cache_refill to test ss_tls_bind_one for C7 class.
- Guarded by HAKMEM_WARM_TLS_BIND_C7 env var and debug build.
- Updated Page Box comments to clarify future TLS Bind Box integration.
 2025-12-05 20:05:11 +09:00
Moe Charm (CI)
45b2ccbe45 Refactor: Extract TLS Bind Box for unified slab binding 
- Created core/box/ss_tls_bind_box.h containing ss_tls_bind_one().
- Refactored superslab_refill() to use the new box.
- Updated signatures to avoid circular dependencies (tiny_self_u32).
- Added future integration points for Warm Pool and Page Box.
 2025-12-05 19:57:30 +09:00
Moe Charm (CI)
093f362231 Add Page Box layer for C7 class optimization 
- Implement tiny_page_box.c/h: per-thread page cache between UC and Shared Pool
- Integrate Page Box into Unified Cache refill path
- Remove legacy SuperSlab implementation (merged into smallmid)
- Add HAKMEM_TINY_PAGE_BOX_CLASSES env var for selective class enabling
- Update bench_random_mixed.c with Page Box statistics

Current status: Implementation safe, no regressions.
Page Box ON/OFF shows minimal difference - pool strategy needs tuning.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-05 15:31:44 +09:00
Moe Charm (CI)
141b121e9c Phase 1: Warm Pool Capacity Increase (16 → 12 with matching threshold) 
Key Changes:
- Reduced static capacity from 16 to 12 SuperSlabs per class
- Fixed prefill threshold from hardcoded 4 to match capacity (12)
- Updated environment variable clamping to [1,12]
- This allows warm pool to actually utilize its full capacity

Performance:
- Baseline (post-unified-cache-opt): 4.76M ops/s
- After Phase 1: 4.84M ops/s
- Improvement: +1.6% (expected +15-20%)

Note: Actual improvement lower than expected because the warm pool
bottleneck is only part of the overall allocation path. Unified cache
optimization (+14.9%) already addressed much of the registry scan overhead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-05 12:16:39 +09:00
Moe Charm (CI)
a04e3ba0e9 Optimize Unified Cache: Batch Freelist Validation + TLS Alignment 
Two complementary optimizations to improve unified cache hot path performance:

1. Batch Freelist Validation (core/front/tiny_unified_cache.c)
   - Remove duplicate per-block freelist validation in release builds
   - Consolidated validation logic into unified_refill_validate_base() function
   - Previously: hak_super_lookup(p) called on EVERY freelist block (~128 blocks)
   - Now: Single validation function at batch start
   - Impact (RELEASE): Eliminates 50-100 cycles per block × 128 = 1,280-2,560 cycles/refill
   - Impact (DEBUG): Full validation still available via unified_refill_validate_base()
   - Safety: Block integrity protected by header magic (0xA0 | class_idx)

2. TLS Unified Cache Alignment (core/front/tiny_unified_cache.h)
   - Add __attribute__((aligned(64))) to TinyUnifiedCache struct
   - Aligns each per-class cache to 64-byte cache line boundary
   - Eliminates false sharing across classes (8 classes × 64B = 512B per thread)
   - Prevents cache line thrashing on concurrent class access
   - Fields stay same size (16B data + 48B padding), no binary compatibility issues
   - Requires clean rebuild due to struct size change (16B → 64B)

Performance Expectations (projected, pending clean build measurement):
- random_mixed (256B working set): +15-20% throughput gain
- tiny_hot: No regression (already cache-friendly)
- tiny_malloc: +3-5% throughput gain

Benchmark Results (after clean rebuild):
- Target: 4.3M → 5.0M ops/s (+17%)
- tiny_hot: Maintain 150M+ ops/s (no regression)

Code Quality:
- ✅ Proper separation of concerns (validation logic centralized)
- ✅ Clean compile-time gating with #if HAKMEM_BUILD_RELEASE
- ✅ Memory-safe (all access patterns unchanged)
- ✅ Maintainable (single source of truth for validation)

Testing Required:
- [ ] Clean rebuild (make clean && make bench_random_mixed_hakmem)
- [ ] Performance measurement with consistent parameters
- [ ] Debug build validation test (ensure corruption detection still works)
- [ ] Multi-threaded correctness (TLS alignment safe for MT)

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: ChatGPT (optimization implementation)
 2025-12-05 11:32:07 +09:00
Moe Charm (CI)
cd3280eee7 Implement MADV_POPULATE_WRITE fix for SuperSlab allocation 
Add support for MADV_POPULATE_WRITE (Linux 5.14+) to force page population
AFTER munmap trimming in SuperSlab fallback path.

Changes:
1. core/box/ss_os_acquire_box.c (lines 171-201):
   - Apply MADV_POPULATE_WRITE after munmap prefix/suffix trim
   - Fallback to explicit page touch for kernels < 5.14
   - Always cleanup suffix region (remove MADV_DONTNEED path)

2. core/superslab_cache.c (lines 111-121):
   - Use MADV_POPULATE_WRITE instead of memset for efficiency
   - Fallback to memset if madvise fails

Testing Results:
- Page faults: Unchanged (~145K per 1M ops)
- Throughput: -2% (4.18M → 4.10M ops/s with HAKMEM_SS_PREFAULT=1)
- Root cause: 97.6% of page faults are from libc memset in initialization,
  not from SuperSlab memory access

Conclusion: MADV_POPULATE_WRITE is effective for SuperSlab memory,
but overall page fault bottleneck comes from TLS/shared pool initialization.
Startup warmup remains the most effective solution (already implemented
in bench_random_mixed.c with +9.5% improvement).

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-05 10:42:47 +09:00
Moe Charm (CI)
1cdc932fca Performance Optimization: Release Build Hygiene (Priority 1-4) 
Implement 4 targeted optimizations for release builds:

1. **Remove freelist validation from release builds** (Priority 1)
   - Guard registry lookup on every freelist node with #if !HAKMEM_BUILD_RELEASE
   - Expected gain: +15-20% throughput (eliminates 30-40% of refill cycles)
   - File: core/front/tiny_unified_cache.c:501-529

2. **Optimize PageFault telemetry** (Priority 2)
   - Already properly gated with HAKMEM_DEBUG_COUNTERS
   - No change needed (verified correct implementation)

3. **Make warm pool stats compile-time gated** (Priority 3)
   - Guard all stats recording with #if HAKMEM_DEBUG_COUNTERS
   - File: core/box/warm_pool_stats_box.h:25-51

4. **Reduce warm pool prefill lock overhead** (Priority 4)
   - Reduced WARM_POOL_PREFILL_BUDGET from 3 to 2 SuperSlabs
   - Balances prefill lock overhead with pool depletion frequency
   - File: core/box/warm_pool_prefill_box.h:28

5. **Disable debug counters by default in release builds** (Supporting)
   - Modified HAKMEM_DEBUG_COUNTERS to auto-detect based on NDEBUG
   - File: core/hakmem_build_flags.h:33-40

Benchmark Results (1M allocations, ws=256):
- Before: 4.02-4.2M ops/s (with diagnostic overhead)
- After: 4.04-4.2M ops/s (release build optimized)
- Warm pool hit rate: Maintained at 55.6%
- No performance regressions detected

Expected Impact After Compilation:
- With -DHAKMEM_BUILD_RELEASE=1 and -DNDEBUG:
  - Freelist validation: compiled out completely
  - Debug counters: compiled out completely
  - Telemetry: compiled out completely
  - Stats recording: compiled out (single (void) statement remains)
  - Expected +15-25% improvement in release builds

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-05 06:16:12 +09:00
Moe Charm (CI)
b6010dd253 Modularize Warm Pool with 3 Box Refactorings - Phase B-3a Complete 
Objective: Clean up warm pool implementation by extracting inline boxes
for statistics, carving, and prefill logic. Achieved full modularity
with zero performance regression using aggressive inline optimization.

Changes:

1. **Legacy Code Removal** (Phase 0)
   - Removed unused static __thread prefill_attempt_count variable
   - Cleaned up duplicate comments
   - Simplified carve failure handling

2. **Warm Pool Statistics Box** (Phase 1)
   - New file: core/box/warm_pool_stats_box.h
   - Inline APIs: warm_pool_record_hit/miss/prefilled()
   - All statistics recording externalized
   - Integrated into unified_cache.c
   - Performance: 0 cost (inlined to direct memory write)

3. **Slab Carving Box** (Phase 2)
   - New file: core/box/slab_carve_box.h
   - Inline API: slab_carve_from_ss()
   - Extracted unified_cache_carve_from_ss() function
   - Now reusable by other refill paths (P0, etc.)
   - Performance: 100% inlined, O(slabs) scan unchanged

4. **Warm Pool Prefill Box** (Phase 3)
   - New file: core/box/warm_pool_prefill_box.h
   - Inline API: warm_pool_do_prefill()
   - Extracted prefill loop with configurable budget
   - WARM_POOL_PREFILL_BUDGET = 3 (tunable)
   - Cold path optimization (only on empty pool)
   - Performance: Cold path cost (non-critical)

Architecture:
- core/front/tiny_unified_cache.c now 40+ lines shorter
- Logic distributed to 3 well-defined boxes
- Each box has single responsibility (SRP)
- Inline compilation preserves hot path performance
- LTO (-flto) enables cross-file inlining

Performance Results:
- 1M allocations: 4.099M ops/s (maintained)
- 5M allocations: 4.046M ops/s (maintained)
- 55.6% warm pool hit rate (unchanged)
- Zero regression on throughput
- All three boxes fully inlined by compiler

Code Quality Improvements:
✅ Removed legacy unused variables
✅ Separated concerns into specialized boxes
✅ Improved readability and maintainability
✅ Preserved performance via aggressive inline
✅ Enabled future reuse (carve box for P0)

Testing:
✅ Compilation: No errors
✅ Functionality: 1M and 5M allocation tests pass
✅ Performance: Baseline maintained
✅ Statistics: Output identical to pre-refactor

Next Phase: Consider similar modularization for:
- Registry scanning (registry_scan_box.h)
- TLS management (tls_management_box.h)
- Cache operations (unified_cache_policy_box.h)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 23:39:02 +09:00
Moe Charm (CI)
5685c2f4c9 Implement Warm Pool Secondary Prefill Optimization (Phase B-2c Complete) 
Problem: Warm pool had 0% hit rate (only 1 hit per 3976 misses) despite being
implemented, causing all cache misses to go through expensive superslab_refill
registry scans.

Root Cause Analysis:
- Warm pool was initialized once and pushed a single slab after each refill
- When that slab was exhausted, it was discarded (not pushed back)
- Next refill would push another single slab, which was immediately exhausted
- Pool would oscillate between 0 and 1 items, yielding 0% hit rate

Solution: Secondary Prefill on Cache Miss
When warm pool becomes empty, we now do multiple superslab_refills and prefill
the pool with 3 additional HOT superlslabs before attempting to carve. This
builds a working set of slabs that can sustain allocation pressure.

Implementation Details:
- Modified unified_cache_refill() cold path to detect empty pool
- Added prefill loop: when pool count == 0, load 3 extra superlslabs
- Store extra slabs in warm pool, keep 1 in TLS for immediate carving
- Track prefill events in g_warm_pool_stats[].prefilled counter

Results (1M Random Mixed 256B allocations):
- Before: C7 hits=1, misses=3976, hit_rate=0.0%
- After:  C7 hits=3929, misses=3143, hit_rate=55.6%
- Throughput: 4.055M ops/s (maintained vs 4.07M baseline)
- Stability: Consistent 55.6% hit rate at 5M allocations (4.102M ops/s)

Performance Impact:
- No regression: throughput remained stable at ~4.1M ops/s
- Registry scan avoided in 55.6% of cache misses (significant savings)
- Warm pool now functioning as intended with strong locality

Configuration:
- TINY_WARM_POOL_MAX_PER_CLASS increased from 4 to 16 to support prefill
- Prefill budget hardcoded to 3 (tunable via env var if needed later)
- All statistics always compiled, ENV-gated printing via HAKMEM_WARM_POOL_STATS=1

Next Steps:
- Monitor for further optimization opportunities (prefill budget tuning)
- Consider adaptive prefill budget based on class-specific hit rates
- Validate at larger allocation counts (10M+ pending registry size fix)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 23:31:54 +09:00
Moe Charm (CI)
4cad395e10 Implement and Test Lazy Zeroing Optimization: Phase 2 Complete 
## Implementation
- Added MADV_DONTNEED when SuperSlab enters LRU cache
- Environment variable: HAKMEM_SS_LAZY_ZERO (default: 1)
- Low-risk, zero-overhead when disabled

## Results: NO MEASURABLE IMPROVEMENT
- Cycles: 70.4M (baseline) vs 70.8M (optimized) = -0.5% (worse!)
- Page faults: 7,674 (no change)
- L1 misses: 717K vs 714K (negligible)

## Key Discovery
The 11.65% clear_page_erms overhead is **kernel-level**, not allocator-level:
- Happens during page faults, not during free
- Can't be selectively deferred for SuperSlab pages
- MADV_DONTNEED syscall overhead cancels benefit
- Result: Zero improvement despite profiling showing 11.65%

## Why Profiling Was Misleading
- Page zeroing shown in profile but not controllable
- Happens globally across all allocators
- Can't isolate which faults are from our code
- Not all profile % are equally optimizable

## Conclusion
Random Mixed 1.06M ops/s appears to be near the practical limit:
- THP: no effect (already tested)
- PREFAULT: +2.6% (measurement noise)
- Lazy zeroing: 0% (syscall overhead cancels benefit)
- Realistic cap: ~1.10-1.15M ops/s (10-15% max possible)

Tiny Hot (89M ops/s) is not comparable - it's an architectural difference.

🐱 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 20:49:21 +09:00
Moe Charm (CI)
cba6f785a1 Add SuperSlab Prefault Box with 4MB MAP_POPULATE bug fix 
New Feature: ss_prefault_box.h
- Box for controlling SuperSlab page prefaulting policy
- ENV: HAKMEM_SS_PREFAULT (0=OFF, 1=POPULATE, 2=TOUCH)
- Default: OFF (safe mode until further optimization)

Bug Fix: 4MB MAP_POPULATE regression
- Problem: Fallback path allocated 4MB (2x size for alignment) with MAP_POPULATE
  causing 52x slower mmap (0.585ms → 30.6ms) and 35% throughput regression
- Solution: Remove MAP_POPULATE from 4MB allocation, apply madvise(MADV_WILLNEED)
  only to the aligned 2MB region after trimming prefix/suffix

Changes:
- core/box/ss_prefault_box.h: New prefault policy box (header-only)
- core/box/ss_allocation_box.c: Integrate prefault box, call ss_prefault_region()
- core/superslab_cache.c: Fix fallback path - no MAP_POPULATE on 4MB,
  always munmap prefix/suffix, use MADV_WILLNEED for 2MB only
- docs/specs/ENV_VARS*.md: Document HAKMEM_SS_PREFAULT

Performance:
- bench_random_mixed: 4.32M ops/s (regression fixed, slight improvement)
- bench_tiny_hot: 157M ops/s with prefault=1 (no crash)

Box Theory:
- OS layer (ss_os_acquire): "how to mmap"
- Prefault Box: "when to page-in"
- Allocation Box: "when to call prefault"

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 20:11:24 +09:00
Moe Charm (CI)
a32d0fafd4 Two-Speed Optimization Part 2: Remove atomic trace counters from hot path 
Performance improvements:
- lock incl instructions completely removed from malloc/free hot paths
- Cache misses reduced from 24.4% → 13.4% of cycles
- Throughput: 85M → 89.12M ops/sec (+4.8% improvement)
- Cycles/op: 48.8 → 48.25 (-1.1%)

Changes in core/box/hak_wrappers.inc.h:
- malloc: Guard g_wrap_malloc_trace_count atomic with #if !HAKMEM_BUILD_RELEASE
- free: Guard g_wrap_free_trace_count and g_free_wrapper_calls with same guard

Debug builds retain full instrumentation via HAK_TRACE.
Release builds execute completely clean hot paths without atomic operations.

Verified via:
- perf report: lock incl instructions gone
- perf stat: cycles/op reduced, cache miss % improved
- objdump: 0 lock instructions in hot paths

Next: Inline unified_cache_refill for additional 3-4 cycles/op improvement

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 19:20:44 +09:00
Moe Charm (CI)
c1c45106da Two-Speed HOT PATH: Guard hak_super_lookup calls with HAKMEM_BUILD_RELEASE 
Phase E2 introduced registry lookup to the hot path, causing 84-88% regression
(70M → 9M ops/sec). This commit restores performance by guarding expensive
hak_super_lookup calls (50-100 cycles each) with conditional compilation.

Key changes:
- tls_sll_box.h push: Full validation in Debug, ss_fast_lookup (O(1)) in Release
- tls_sll_box.h pop: Registry validation in Debug, trust list structure in Release
- tiny_free_fast_v2.inc.h: Header/meta cross-check Debug-only
- malloc_tiny_fast.h: SuperSlab registration check Debug-only

Performance improvement:
- Release build: 2.9M → 87-88M ops/sec (30x improvement)
- Restored to historical UNIFIED-HEADER peak (70-80M range)

Release builds trust:
- Header magic (0xA0) as sufficient allocation origin validation
- TLS SLL linked list structure integrity
- Header-based class_idx classification

Debug builds maintain full validation with expensive registry lookups.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 18:53:04 +09:00
Moe Charm (CI)
860991ee50 Performance Measurement Framework: Unified Cache, TLS SLL, Shared Pool Analysis 
## Summary

Implemented production-grade measurement infrastructure to quantify top 3 bottlenecks:
- Unified cache hit/miss rates + refill cost
- TLS SLL usage patterns
- Shared pool lock contention distribution

## Changes

### 1. Unified Cache Metrics (tiny_unified_cache.h/c)
- Added atomic counters:
  - g_unified_cache_hits_global: successful cache pops
  - g_unified_cache_misses_global: refill triggers
  - g_unified_cache_refill_cycles_global: refill cost in CPU cycles (rdtsc)
- Instrumented `unified_cache_pop_or_refill()` to count hits
- Instrumented `unified_cache_refill()` with cycle measurement
- ENV-gated: HAKMEM_MEASURE_UNIFIED_CACHE=1 (default: off)
- Added unified_cache_print_measurements() output function

### 2. TLS SLL Metrics (tls_sll_box.h)
- Added atomic counters:
  - g_tls_sll_push_count_global: total pushes
  - g_tls_sll_pop_count_global: successful pops
  - g_tls_sll_pop_empty_count_global: empty list conditions
- Instrumented push/pop paths
- Added tls_sll_print_measurements() output function

### 3. Shared Pool Contention (hakmem_shared_pool_acquire.c)
- Added atomic counters:
  - g_sp_stage2_lock_acquired_global: Stage 2 locks
  - g_sp_stage3_lock_acquired_global: Stage 3 allocations
  - g_sp_alloc_lock_contention_global: total lock acquisitions
- Instrumented all pthread_mutex_lock calls in hot paths
- Added shared_pool_print_measurements() output function

### 4. Benchmark Integration (bench_random_mixed.c)
- Called all 3 print functions after benchmark loop
- Functions active only when HAKMEM_MEASURE_UNIFIED_CACHE=1 set

## Design Principles

- **Zero overhead when disabled**: Inline checks with __builtin_expect hints
- **Atomic relaxed memory order**: Minimal synchronization overhead
- **ENV-gated**: Single flag controls all measurements
- **Production-safe**: Compiles in release builds, no functional changes

## Usage

```bash
HAKMEM_MEASURE_UNIFIED_CACHE=1 ./bench_allocators_hakmem bench_random_mixed_hakmem 1000000 256 42
```

Output (when enabled):
```
========================================
Unified Cache Statistics
========================================
Hits:        1234567
Misses:      56789
Hit Rate:    95.6%
Avg Refill Cycles: 1234

========================================
TLS SLL Statistics
========================================
Total Pushes:     1234567
Total Pops:       345678
Pop Empty Count:  12345
Hit Rate:         98.8%

========================================
Shared Pool Contention Statistics
========================================
Stage 2 Locks:    123456 (33%)
Stage 3 Locks:    234567 (67%)
Total Contention: 357 locks per 1M ops
```

## Next Steps

1. **Enable measurements** and run benchmarks to gather data
2. **Analyze miss rates**: Which bottleneck dominates?
3. **Profile hottest stage**: Focus optimization on top contributor
4. Possible targets:
   - Increase unified cache capacity if miss rate >5%
   - Profile if TLS SLL is unused (potential legacy code removal)
   - Analyze if Stage 2 lock can be replaced with CAS

## Makefile Updates

Added core/box/tiny_route_box.o to:
- OBJS_BASE (test build)
- SHARED_OBJS (shared library)
- BENCH_HAKMEM_OBJS_BASE (benchmark)
- TINY_BENCH_OBJS_BASE (tiny benchmark)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 18:26:39 +09:00
Moe Charm (CI)
d5e6ed535c P-Tier + Tiny Route Policy: Aggressive Superslab Management + Safe Routing 
## Phase 1: Utilization-Aware Superslab Tiering (案B実装済)

- Add ss_tier_box.h: Classify SuperSlabs into HOT/DRAINING/FREE based on utilization
  - HOT (>25%): Accept new allocations
  - DRAINING (≤25%): Drain only, no new allocs
  - FREE (0%): Ready for eager munmap

- Enhanced shared_pool_release_slab():
  - Check tier transition after each slab release
  - If tier→FREE: Force remaining slots to EMPTY and call superslab_free() immediately
  - Bypasses LRU cache to prevent registry bloat from accumulating DRAINING SuperSlabs

- Test results (bench_random_mixed_hakmem):
  - 1M iterations: ✅ ~1.03M ops/s (previously passed)
  - 10M iterations: ✅ ~1.15M ops/s (previously: registry full error)
  - 50M iterations: ✅ ~1.08M ops/s (stress test)

## Phase 2: Tiny Front Routing Policy (新規Box)

- Add tiny_route_box.h/c: Single 8-byte table for class→routing decisions
  - ROUTE_TINY_ONLY: Tiny front exclusive (no fallback)
  - ROUTE_TINY_FIRST: Try Tiny, fallback to Pool if fails
  - ROUTE_POOL_ONLY: Skip Tiny entirely

- Profiles via HAKMEM_TINY_PROFILE ENV:
  - "hot": C0-C3=TINY_ONLY, C4-C6=TINY_FIRST, C7=POOL_ONLY
  - "conservative" (default): All TINY_FIRST
  - "off": All POOL_ONLY (disable Tiny)
  - "full": All TINY_ONLY (microbench mode)

- A/B test results (ws=256, 100k ops random_mixed):
  - Default (conservative): ~2.90M ops/s
  - hot: ~2.65M ops/s (more conservative)
  - off: ~2.86M ops/s
  - full: ~2.98M ops/s (slightly best)

## Design Rationale

### Registry Pressure Fix (案B)
- Problem: DRAINING tier SS occupied registry indefinitely
- Solution: When total_active_blocks→0, immediately free to clear registry slot
- Result: No more "registry full" errors under stress

### Routing Policy Box (新)
- Problem: Tiny front optimization scattered across ENV/branches
- Solution: Centralize routing in single table, select profiles via ENV
- Benefit: Safe A/B testing without touching hot path code
- Future: Integrate with RSS budget/learning layers for dynamic profile switching

## Next Steps (性能最適化)
- Profile Tiny front internals (TLS SLL, FastCache, Superslab backend latency)
- Identify bottleneck between current ~2.9M ops/s and mimalloc ~100M ops/s
- Consider:
  - Reduce shared pool lock contention
  - Optimize unified cache hit rate
  - Streamline Superslab carving logic

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 18:01:25 +09:00
Moe Charm (CI)
984cca41ef P0 Optimization: Shared Pool fast path with O(1) metadata lookup 
Performance Results:
- Throughput: 2.66M ops/s → 3.8M ops/s (+43% improvement)
- sp_meta_find_or_create: O(N) linear scan → O(1) direct pointer
- Stage 2 metadata scan: 100% → 10-20% (80-90% reduction via hints)

Core Optimizations:

1. O(1) Metadata Lookup (superslab_types.h)
   - Added `shared_meta` pointer field to SuperSlab struct
   - Eliminates O(N) linear search through ss_metadata[] array
   - First access: O(N) scan + cache | Subsequent: O(1) direct return

2. sp_meta_find_or_create Fast Path (hakmem_shared_pool.c)
   - Check cached ss->shared_meta first before linear scan
   - Cache pointer after successful linear scan for future lookups
   - Reduces 7.8% CPU hotspot to near-zero for hot paths

3. Stage 2 Class Hints Fast Path (hakmem_shared_pool_acquire.c)
   - Try class_hints[class_idx] FIRST before full metadata scan
   - Uses O(1) ss->shared_meta lookup for hint validation
   - __builtin_expect() for branch prediction optimization
   - 80-90% of acquire calls now skip full metadata scan

4. Proper Initialization (ss_allocation_box.c)
   - Initialize shared_meta = NULL in superslab_allocate()
   - Ensures correct NULL-check semantics for new SuperSlabs

Additional Improvements:
- Updated ptr_trace and debug ring for release build efficiency
- Enhanced ENV variable documentation and analysis
- Added learner_env_box.h for configuration management
- Various Box optimizations for reduced overhead

Thread Safety:
- All atomic operations use correct memory ordering
- shared_meta cached under mutex protection
- Lock-free Stage 2 uses proper CAS with acquire/release semantics

Testing:
- Benchmark: 1M iterations, 3.8M ops/s stable
- Build: Clean compile RELEASE=0 and RELEASE=1
- No crashes, memory leaks, or correctness issues

Next Optimization Candidates:
- P1: Per-SuperSlab free slot bitmap for O(1) slot claiming
- P2: Reduce Stage 2 critical section size
- P3: Page pre-faulting (MAP_POPULATE)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-12-04 16:21:54 +09:00