tomoaki/hakmem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0bc33dc4f5ed508bace78ae96d6bd44ef2d92205
hakmem/CURRENT_TASK.md

52 lines
3.0 KiB
Markdown
Raw Normal View History

Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
# Current Task: Phase 9-2 Refactoring (Complete)
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
docs: Update CURRENT_TASK.md and claude.md for Phase 8 completion Phase 8 Complete: BenchFast crash root cause fixes Documentation updates: 1. CURRENT_TASK.md: - Phase 8 complete (TLS→Atomic + Header write fixes) - 箱理論 root cause analysis (3 critical bugs) - Next phase recommendations (Option C: BenchFast pool expansion) - Detailed technical explanations for each layer 2. .claude/claude.md: - Phase 8 achievement summary - 箱理論 4-principle validation - Commit references (191e65983, da8f4d2c8) Key Fixes Documented: - TLS→Atomic: Cross-thread guard variable (pthread_once bug) - Header Write: Direct write bypasses P3 optimization (free routing) - Infrastructure Isolation: __libc_calloc for cache arrays - Design Fix: Removed unified_cache_init() call 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 05:50:43 +09:00
**Date**: 2025-11-30
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
**Status**: **COMPLETE** (Phase 9-2 & Refactoring)
**Goal**: SuperSlab Unified Management, Stability Fixes, and Code Refactoring
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
---
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
## Phase 9-2 Achievements (Completed)
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
1. **Critical Fixes (Deadlock & OOM)**
* **Deadlock**: `shared_pool_acquire_slab` now releases `alloc_lock` before calling `superslab_allocate` (via `sp_internal_allocate_superslab`), preventing lock inversion with `g_super_reg_lock`.
* **OOM**: Enabled `HAKMEM_TINY_USE_SUPERSLAB=1` by default in `hakmem_build_flags.h`, ensuring fallback to Legacy Backend when Shared Pool hits soft cap.
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
2. **SuperSlab Management Unification**
* **Unified Entry**: `sp_internal_allocate_superslab` helper introduced to manage safe allocation flow.
* **Unified Free**: `remove_superslab_from_legacy_head` implemented to safely remove pointers from legacy lists when freeing via Shared Pool.
Phase 5: Documentation & Task Update (COMPLETE) Phase 5 Mid/Large Allocation Optimization complete with major success. Achievement: - Mid MT allocations (1KB-8KB): +28.9x improvement (1.49M → 41.0M ops/s) - vs System malloc: 1.53x faster (41.0 vs 26.8 M ops/s) - Mid Free Route Box: Fixed 19x free() slowdown via dual-registry routing Files: - PHASE5_COMPLETION_REPORT.md (NEW) - Full completion report with technical details - CURRENT_TASK.md - Updated with Phase 5 completion and next phase options Completed Steps: - Step 1: Mid MT Verification (range bug identified) - Step 2: Mid Free Route Box (+28.9x improvement) - Step 3: Mid/Large Config Box (future workload infrastructure) - Step 4: Deferred (MT workload needed) - Step 5: Documentation (this commit) Next Phase Options: - Option A: Investigate bench_random_mixed regression - Option B: PGO re-enablement (recommended, +6.25% proven) - Option C: Expand Tiny Front Config Box - Option D: Production readiness & benchmarking - Option E: Multi-threaded optimization See PHASE5_COMPLETION_REPORT.md for full technical details and CURRENT_TASK.md for next phase recommendations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 14:46:54 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
3. **Code Refactoring (Split `hakmem_shared_pool.c`)**
* **Split Strategy**: Divided the monolithic `core/hakmem_shared_pool.c` (1400+ lines) into logical modules:
* `core/hakmem_shared_pool.c`: Initialization, stats, and common helpers.
* `core/hakmem_shared_pool_acquire.c`: Allocation logic (`shared_pool_acquire_slab` and Stage 0.5-3).
* `core/hakmem_shared_pool_release.c`: Deallocation logic (`shared_pool_release_slab`).
* `core/hakmem_shared_pool_internal.h`: Internal shared definitions and prototypes.
* **Makefile**: Updated to compile and link the new files.
* **Cleanups**: Removed unused "L0 Cache" experimental code and fixed incorrect function names (`superslab_alloc` -> `superslab_allocate`).
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
Update CURRENT_TASK.md: Phase 7 complete (+54.2% improvement!)
2025-11-29 16:20:58 +09:00
---
docs: Start Phase 5 - Mid/Large Allocation Optimization Update CURRENT_TASK.md with Phase 5 roadmap: - Goal: +10-26% improvement (57.2M → 63-72M ops/s) - Strategy: Fix allocation gap + Config Box + Mid MT optimization - Duration: 12 days / 2 weeks Phase 5 Steps: 1. Mid MT Verification (2 days) 2. Allocation Gap Elimination (3 days) - Priority 1 3. Mid/Large Config Box (3 days) 4. Mid Registry Pre-allocation (2 days) 5. Documentation & Benchmark (2 days) Critical Issue Found: - 1KB-8KB allocations fall through to mmap() when ACE disabled - Impact: 1000-5000x slower than O(1) allocation - Fix: Route through existing Mid MT allocator Phase 4 Complete: - Result: 53.3M → 57.2M ops/s (+7.3%) - PGO deferred to final optimization phase 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 12:30:29 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
## Next Phase Candidates (Handover from Phase 9-2)
Phase 4-Step1: Add PGO workflow automation (+6.25% performance) Implemented automated Profile-Guided Optimization workflow using Box pattern: Performance Improvement: - Baseline: 57.0 M ops/s - PGO-optimized: 60.6 M ops/s - Gain: +6.25% (within expected +5-10% range) Implementation: 1. scripts/box/pgo_tiny_profile_config.sh - 5 representative workloads 2. scripts/box/pgo_tiny_profile_box.sh - Automated profile collection 3. Makefile PGO targets: - pgo-tiny-profile: Build instrumented binaries - pgo-tiny-collect: Collect .gcda profile data - pgo-tiny-build: Build optimized binaries - pgo-tiny-full: Complete workflow (profile → collect → build → test) 4. Makefile help target: Added PGO instructions for discoverability Design: - Box化: Single responsibility, clear contracts - Deterministic: Fixed seeds (42) for reproducibility - Safe: Validation, error detection, timeout protection (30s/workload) - Observable: Progress reporting, .gcda verification (33 files generated) Workload Coverage: - Random mixed: 3 working set sizes (128/256/512 slots) - Tiny hot: 2 size classes (16B/64B) - Total: 5 workloads covering hot/cold paths Documentation: - PHASE4_STEP1_COMPLETE.md - Completion report - CURRENT_TASK.md - Phase 4 roadmap (Step 1 complete ✓) - docs/design/PHASE4_TINY_FRONT_BOX_DESIGN.md - Complete Phase 4 design Next: Phase 4-Step2 (Hot/Cold Path Box, target +10-15%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 11:28:38 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
### 1. Soft Cap (Policy) Tuning
* **Issue**: Medium Working Sets (8192) hit the Shared Pool "Soft Cap" easily, causing frequent fallbacks and performance degradation.
* **Action**: Review `hakmem_policy.c` and adjust `tiny_cap` or improve dynamic adjustment logic.
Phase 4-Step1: Add PGO workflow automation (+6.25% performance) Implemented automated Profile-Guided Optimization workflow using Box pattern: Performance Improvement: - Baseline: 57.0 M ops/s - PGO-optimized: 60.6 M ops/s - Gain: +6.25% (within expected +5-10% range) Implementation: 1. scripts/box/pgo_tiny_profile_config.sh - 5 representative workloads 2. scripts/box/pgo_tiny_profile_box.sh - Automated profile collection 3. Makefile PGO targets: - pgo-tiny-profile: Build instrumented binaries - pgo-tiny-collect: Collect .gcda profile data - pgo-tiny-build: Build optimized binaries - pgo-tiny-full: Complete workflow (profile → collect → build → test) 4. Makefile help target: Added PGO instructions for discoverability Design: - Box化: Single responsibility, clear contracts - Deterministic: Fixed seeds (42) for reproducibility - Safe: Validation, error detection, timeout protection (30s/workload) - Observable: Progress reporting, .gcda verification (33 files generated) Workload Coverage: - Random mixed: 3 working set sizes (128/256/512 slots) - Tiny hot: 2 size classes (16B/64B) - Total: 5 workloads covering hot/cold paths Documentation: - PHASE4_STEP1_COMPLETE.md - Completion report - CURRENT_TASK.md - Phase 4 roadmap (Step 1 complete ✓) - docs/design/PHASE4_TINY_FRONT_BOX_DESIGN.md - Complete Phase 4 design Next: Phase 4-Step2 (Hot/Cold Path Box, target +10-15%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 11:28:38 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
### 2. Fast Path Optimization
* **Issue**: Small Working Sets (256) show 70-88% performance vs SysAlloc due to lock/call overhead. Refactoring caused a slight dip (15%), highlighting the need for optimization.
* **Action**: Re-implement a lightweight L0 Cache or optimize the lock-free path in Shared Pool for hot-path performance. Consider inlining hot helpers again via header-only implementations if needed.
Phase 4-Step1: Add PGO workflow automation (+6.25% performance) Implemented automated Profile-Guided Optimization workflow using Box pattern: Performance Improvement: - Baseline: 57.0 M ops/s - PGO-optimized: 60.6 M ops/s - Gain: +6.25% (within expected +5-10% range) Implementation: 1. scripts/box/pgo_tiny_profile_config.sh - 5 representative workloads 2. scripts/box/pgo_tiny_profile_box.sh - Automated profile collection 3. Makefile PGO targets: - pgo-tiny-profile: Build instrumented binaries - pgo-tiny-collect: Collect .gcda profile data - pgo-tiny-build: Build optimized binaries - pgo-tiny-full: Complete workflow (profile → collect → build → test) 4. Makefile help target: Added PGO instructions for discoverability Design: - Box化: Single responsibility, clear contracts - Deterministic: Fixed seeds (42) for reproducibility - Safe: Validation, error detection, timeout protection (30s/workload) - Observable: Progress reporting, .gcda verification (33 files generated) Workload Coverage: - Random mixed: 3 working set sizes (128/256/512 slots) - Tiny hot: 2 size classes (16B/64B) - Total: 5 workloads covering hot/cold paths Documentation: - PHASE4_STEP1_COMPLETE.md - Completion report - CURRENT_TASK.md - Phase 4 roadmap (Step 1 complete ✓) - docs/design/PHASE4_TINY_FRONT_BOX_DESIGN.md - Complete Phase 4 design Next: Phase 4-Step2 (Hot/Cold Path Box, target +10-15%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 11:28:38 +09:00
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
### 3. Legacy Backend Removal
* **Issue**: Legacy Backend (`g_superslab_heads`) is still kept for fallback but causes complexity.
* **Action**: Plan complete removal of `g_superslab_heads`, migrating all management to Shared Pool.
Phase 4-Step1: Add PGO workflow automation (+6.25% performance) Implemented automated Profile-Guided Optimization workflow using Box pattern: Performance Improvement: - Baseline: 57.0 M ops/s - PGO-optimized: 60.6 M ops/s - Gain: +6.25% (within expected +5-10% range) Implementation: 1. scripts/box/pgo_tiny_profile_config.sh - 5 representative workloads 2. scripts/box/pgo_tiny_profile_box.sh - Automated profile collection 3. Makefile PGO targets: - pgo-tiny-profile: Build instrumented binaries - pgo-tiny-collect: Collect .gcda profile data - pgo-tiny-build: Build optimized binaries - pgo-tiny-full: Complete workflow (profile → collect → build → test) 4. Makefile help target: Added PGO instructions for discoverability Design: - Box化: Single responsibility, clear contracts - Deterministic: Fixed seeds (42) for reproducibility - Safe: Validation, error detection, timeout protection (30s/workload) - Observable: Progress reporting, .gcda verification (33 files generated) Workload Coverage: - Random mixed: 3 working set sizes (128/256/512 slots) - Tiny hot: 2 size classes (16B/64B) - Total: 5 workloads covering hot/cold paths Documentation: - PHASE4_STEP1_COMPLETE.md - Completion report - CURRENT_TASK.md - Phase 4 roadmap (Step 1 complete ✓) - docs/design/PHASE4_TINY_FRONT_BOX_DESIGN.md - Complete Phase 4 design Next: Phase 4-Step2 (Hot/Cold Path Box, target +10-15%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 11:28:38 +09:00
---
Refactor: Split monolithic hakmem_shared_pool.c into acquire/release modules - Split core/hakmem_shared_pool.c into acquire/release modules for maintainability. - Introduced core/hakmem_shared_pool_internal.h for shared internal API. - Fixed incorrect function name usage (superslab_alloc -> superslab_allocate). - Increased SUPER_REG_SIZE to 1M to support large working sets (Phase 9-2 fix). - Updated Makefile. - Verified with benchmarks.
2025-11-30 18:11:08 +09:00
## Current Status
* **Build**: Passing (Clean build verified).
* **Benchmarks**:
* `HAKMEM_TINY_SS_SHARED=1` (Normal): ~20.0 M ops/s (working, fallback active).
* `HAKMEM_TINY_SS_SHARED=2` (Strict): ~20.3 M ops/s (working, OOMs on soft cap as expected).
* **Pending**: Selection of next focus area.
Reference in New Issue Copy Permalink