03ba62df4d
Phase 23 Unified Cache + PageFaultTelemetry generalization: Mid/VM page-fault bottleneck identified
...
Summary:
- Phase 23 Unified Cache: +30% improvement (Random Mixed 256B: 18.18M → 23.68M ops/s)
- PageFaultTelemetry: Extended to generic buckets (C0-C7, MID, L25, SSM)
- Measurement-driven decision: Mid/VM page-faults (80-100K) >> Tiny (6K) → prioritize Mid/VM optimization
Phase 23 Changes:
1. Unified Cache implementation (core/front/tiny_unified_cache.{c,h})
- Direct SuperSlab carve (TLS SLL bypass)
- Self-contained pop-or-refill pattern
- ENV: HAKMEM_TINY_UNIFIED_CACHE=1, HAKMEM_TINY_UNIFIED_C{0-7}=128
2. Fast path pruning (tiny_alloc_fast.inc.h, tiny_free_fast_v2.inc.h)
- Unified ON → direct cache access (skip all intermediate layers)
- Alloc: unified_cache_pop_or_refill() → immediate fail to slow
- Free: unified_cache_push() → fallback to SLL only if full
PageFaultTelemetry Changes:
3. Generic bucket architecture (core/box/pagefault_telemetry_box.{c,h})
- PF_BUCKET_{C0-C7, MID, L25, SSM} for domain-specific measurement
- Integration: hak_pool_try_alloc(), l25_alloc_new_run(), shared_pool_allocate_superslab_unlocked()
4. Measurement results (Random Mixed 500K / 256B):
- Tiny C2-C7: 2-33 pages, high reuse (64-3.8 touches/page)
- SSM: 512 pages (initialization footprint)
- MID/L25: 0 (unused in this workload)
- Mid/Large VM benchmarks: 80-100K page-faults (13-16x higher than Tiny)
Ring Cache Enhancements:
5. Hot Ring Cache (core/front/tiny_ring_cache.{c,h})
- ENV: HAKMEM_TINY_HOT_RING_ENABLE=1, HAKMEM_TINY_HOT_RING_C{0-7}=size
- Conditional compilation cleanup
Documentation:
6. Analysis reports
- RANDOM_MIXED_BOTTLENECK_ANALYSIS.md: Page-fault breakdown
- RANDOM_MIXED_SUMMARY.md: Phase 23 summary
- RING_CACHE_ACTIVATION_GUIDE.md: Ring cache usage
- CURRENT_TASK.md: Updated with Phase 23 results and Phase 24 plan
Next Steps (Phase 24):
- Target: Mid/VM PageArena/HotSpanBox (page-fault reduction 80-100K → 30-40K)
- Tiny SSM optimization deferred (low ROI, ~6K page-faults already optimal)
- Expected improvement: +30-50% for Mid/Large workloads
Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-17 02:47:58 +09:00
9472ee90c9
Fix: Larson multi-threaded crash - 3 critical race conditions in SharedSuperSlabPool
...
Root Cause Analysis (via Task agent investigation):
Larson benchmark crashed with SEGV due to 3 separate race conditions between
lock-free Stage 2 readers and mutex-protected writers in shared_pool_acquire_slab().
Race Condition 1: Non-Atomic Counter
- **Problem**: `ss_meta_count` was `uint32_t` (non-atomic) but read atomically via cast
- **Impact**: Thread A reads partially-updated count, accesses uninitialized metadata[N]
- **Fix**: Changed to `_Atomic uint32_t`, use memory_order_release/acquire
Race Condition 2: Non-Atomic Pointer
- **Problem**: `meta->ss` was plain pointer, read lock-free but freed under mutex
- **Impact**: Thread A loads `meta->ss` after Thread B frees SuperSlab → use-after-free
- **Fix**: Changed to `_Atomic(SuperSlab*)`, set NULL before free, check for NULL
Race Condition 3: realloc() vs Lock-Free Readers (CRITICAL)
- **Problem**: `sp_meta_ensure_capacity()` used `realloc()` which MOVES the array
- **Impact**: Thread B reallocs `ss_metadata`, Thread A accesses OLD (freed) array
- **Fix**: **Removed realloc entirely** - use fixed-size array `ss_metadata[2048]`
Fixes Applied:
1. **core/hakmem_shared_pool.h** (Line 53, 125-126):
- `SuperSlab* ss` → `_Atomic(SuperSlab*) ss`
- `uint32_t ss_meta_count` → `_Atomic uint32_t ss_meta_count`
- `SharedSSMeta* ss_metadata` → `SharedSSMeta ss_metadata[MAX_SS_METADATA_ENTRIES]`
- Removed `ss_meta_capacity` (no longer needed)
2. **core/hakmem_shared_pool.c** (Lines 223-233, 248-287, 577, 631-635, 812-815, 872):
- **sp_meta_ensure_capacity()**: Replaced realloc with capacity check
- **sp_meta_find_or_create()**: atomic_load/store for count and ss pointer
- **Stage 1 (line 577)**: atomic_load for meta->ss
- **Stage 2 (line 631-635)**: atomic_load with NULL check + skip
- **shared_pool_release_slab()**: atomic_store(NULL) BEFORE superslab_free()
- All metadata searches: atomic_load for consistency
Memory Ordering:
- **Release** (line 285): `atomic_fetch_add(&ss_meta_count, 1, memory_order_release)`
→ Publishes all metadata[N] writes before count increment is visible
- **Acquire** (line 620, 631): `atomic_load(..., memory_order_acquire)`
→ Synchronizes-with release, ensures initialized metadata is seen
- **Release** (line 872): `atomic_store(&meta->ss, NULL, memory_order_release)`
→ Prevents Stage 2 from seeing dangling pointer
Test Results:
- **Before**: SEGV crash (1 thread, 2 threads, any iteration count)
- **After**: No crashes, stable execution
- 1 thread: 266K ops/sec (stable, no SEGV)
- 2 threads: 193K ops/sec (stable, no SEGV)
- Warning: `[SP_META_CAPACITY_ERROR] Exceeded MAX_SS_METADATA_ENTRIES=2048`
→ Non-fatal, indicates metadata recycling needed (future optimization)
Known Limitation:
- Fixed array size (2048) may be insufficient for extreme workloads
- Workaround: Increase MAX_SS_METADATA_ENTRIES if needed
- Proper solution: Implement metadata recycling when SuperSlabs are freed
Performance Note:
- Larson still slow (~200K ops/sec vs System 20M ops/sec, 100x slower)
- This is due to lock contention (separate issue, not race condition)
- Crash bug is FIXED, performance optimization is next step
Related Issues:
- Original report: Commit 93cc23450🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-14 23:16:54 +09:00
93cc234505
Fix: 500K iteration SEGV - node pool exhaustion + deadlock
...
Root cause analysis (via Task agent investigation):
- Node pool (512 nodes/class) exhausts at ~500K iterations
- Two separate issues identified:
1. Deadlock in sp_freelist_push_lockfree (FREE path)
2. Node pool exhaustion triggering stack corruption (ALLOC path)
Fixes applied:
1. Deadlock fix (core/hakmem_shared_pool.c:382-387):
- Removed recursive pthread_mutex_lock/unlock in fallback path
- Caller (shared_pool_release_slab:772) already holds lock
- Prevents deadlock on non-recursive mutex
2. Node pool expansion (core/hakmem_shared_pool.h:77):
- Increased MAX_FREE_NODES_PER_CLASS from 512 to 4096
- Supports 500K+ iterations without exhaustion
- Prevents stack corruption in hak_tiny_alloc_slow()
Test results:
- Before: SEGV at 500K with "Node pool exhausted for class 7"
- After: 9.44M ops/s, stable, no warnings, no crashes
Note: This fixes Mid-Large allocator's SP-SLOT Box, not Phase B C23 code.
Phase B (TinyFrontC23Box) remains stable and unaffected.
🤖 Generated with Claude Code (https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-14 19:47:40 +09:00
ec453d67f2
Mid-Large Phase 12 Complete + P0-5 Lock-Free Stage 2
...
**Phase 12 第1ラウンド完了** ✅
- 0.24M → 2.39M ops/s (8T, **+896%**)
- SEGFAULT → Zero crashes (**100% → 0%**)
- futex: 209 → 10 calls (**-95%**)
**P0-5: Lock-Free Stage 2 (Slot Claiming)**
- Atomic SlotState: `_Atomic SlotState state`
- sp_slot_claim_lockfree(): CAS-based UNUSED→ACTIVE transition
- acquire_slab() Stage 2: Lock-free claiming (mutex only for metadata)
- Result: 2.34M → 2.39M ops/s (+2.5% @ 8T)
**Implementation**:
- core/hakmem_shared_pool.h: Atomic SlotState definition
- core/hakmem_shared_pool.c:
- sp_slot_claim_lockfree() (+40 lines)
- Atomic helpers: sp_slot_find_unused/mark_active/mark_empty
- Stage 2 lock-free integration
- Verified via debug logs: STAGE2_LOCKFREE claiming works
**Reports**:
- MID_LARGE_P0_PHASE_REPORT.md: P0-0 to P0-4 comprehensive summary
- MID_LARGE_FINAL_AB_REPORT.md: Complete Phase 12 A/B comparison (17KB)
- Performance evolution table
- Lock contention analysis - Lessons learned
- File inventory
**Tiny Baseline Measurement** 📊
- System malloc: 82.9M ops/s (256B)
- HAKMEM: 8.88M ops/s (256B)
- **Gap: 9.3x slower** (target for next phase)
**Next**: Tiny allocator optimization (drain interval, front cache, perf profile)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-14 16:51:53 +09:00
9830237d56
Phase 12: SP-SLOT Box data structures (Task SP-1)
...
Added per-slot state management for Shared SuperSlab Pool optimization.
Problem:
- Current: 1 SuperSlab mixes multiple classes (C0-C7)
- SuperSlab freed only when ALL classes empty (active_slabs==0)
- Result: SuperSlabs rarely freed, LRU cache unused
Solution: SP-SLOT Box
- Track each slab slot state: UNUSED/ACTIVE/EMPTY
- Per-class free slot lists for efficient reuse
- Free SuperSlab only when ALL slots empty
New Structures:
1. SlotState enum - Per-slot state (UNUSED/ACTIVE/EMPTY)
2. SharedSlot - Per-slot metadata (state, class_idx, slab_idx)
3. SharedSSMeta - Per-SuperSlab slot array management
4. FreeSlotList - Per-class free slot lists
Extended SharedSuperSlabPool:
- free_slots[TINY_NUM_CLASSES_SS] - Per-class lists
- ss_metadata[] - SuperSlab metadata array
Next Steps:
- Task SP-2: Implement 3-stage acquire_slab logic
- Task SP-3: Convert release_slab to slot-based
- Expected: Significant mmap/munmap reduction
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-14 07:59:33 +09:00
03df05ec75
Phase 12: Shared SuperSlab Pool implementation (WIP - runtime crash)
...
## Summary
Implemented Phase 12 Shared SuperSlab Pool (mimalloc-style) to address
SuperSlab allocation churn (877 SuperSlabs → 100-200 target).
## Implementation (ChatGPT + Claude)
1. **Metadata changes** (superslab_types.h):
- Added class_idx to TinySlabMeta (per-slab dynamic class)
- Removed size_class from SuperSlab (no longer per-SuperSlab)
- Changed owner_tid (16-bit) → owner_tid_low (8-bit)
2. **Shared Pool** (hakmem_shared_pool.{h,c}):
- Global pool shared by all size classes
- shared_pool_acquire_slab() - Get free slab for class_idx
- shared_pool_release_slab() - Return slab when empty
- Per-class hints for fast path optimization
3. **Integration** (23 files modified):
- Updated all ss->size_class → meta->class_idx
- Updated all meta->owner_tid → meta->owner_tid_low
- superslab_refill() now uses shared pool
- Free path releases empty slabs back to pool
4. **Build system** (Makefile):
- Added hakmem_shared_pool.o to OBJS_BASE and TINY_BENCH_OBJS_BASE
## Status: ⚠️ Build OK, Runtime CRASH
**Build**: ✅ SUCCESS
- All 23 files compile without errors
- Only warnings: superslab_allocate type mismatch (legacy code)
**Runtime**: ❌ SEGFAULT
- Crash location: sll_refill_small_from_ss()
- Exit code: 139 (SIGSEGV)
- Test case: ./bench_random_mixed_hakmem 1000 256 42
## Known Issues
1. **SEGFAULT in refill path** - Likely shared_pool_acquire_slab() issue
2. **Legacy superslab_allocate()** still exists (type mismatch warning)
3. **Remaining TODOs** from design doc:
- SuperSlab physical layout integration
- slab_handle.h cleanup
- Remove old per-class head implementation
## Next Steps
1. Debug SEGFAULT (gdb backtrace shows sll_refill_small_from_ss)
2. Fix shared_pool_acquire_slab() or superslab_init_slab()
3. Basic functionality test (1K → 100K iterations)
4. Measure SuperSlab count reduction (877 → 100-200)
5. Performance benchmark (+650-860% expected)
## Files Changed (25 files)
core/box/free_local_box.c
core/box/free_remote_box.c
core/box/front_gate_classifier.c
core/hakmem_super_registry.c
core/hakmem_tiny.c
core/hakmem_tiny_bg_spill.c
core/hakmem_tiny_free.inc
core/hakmem_tiny_lifecycle.inc
core/hakmem_tiny_magazine.c
core/hakmem_tiny_query.c
core/hakmem_tiny_refill.inc.h
core/hakmem_tiny_superslab.c
core/hakmem_tiny_superslab.h
core/hakmem_tiny_tls_ops.h
core/slab_handle.h
core/superslab/superslab_inline.h
core/superslab/superslab_types.h
core/tiny_debug.h
core/tiny_free_fast.inc.h
core/tiny_free_magazine.inc.h
core/tiny_remote.c
core/tiny_superslab_alloc.inc.h
core/tiny_superslab_free.inc.h
Makefile
## New Files (3 files)
PHASE12_SHARED_SUPERSLAB_POOL_DESIGN.md
core/hakmem_shared_pool.c
core/hakmem_shared_pool.h
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
Co-Authored-By: ChatGPT <chatgpt@openai.com >
2025-11-13 16:33:03 +09:00
