815 lines
27 KiB
Markdown
815 lines
27 KiB
Markdown
|
|
# HAKMEM Box Theory Refactoring Analysis
|
||
|
|
|
||
|
|
**Date**: 2025-11-08
|
||
|
|
**Analyst**: Claude Task Agent (Ultrathink Mode)
|
||
|
|
**Focus**: Phase 2 additions, Phase 6-2.x bug locations, Large files (>500 lines)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Executive Summary
|
||
|
|
|
||
|
|
This analysis identifies **10 high-priority refactoring opportunities** to improve code maintainability, testability, and debuggability using Box Theory principles. The analysis focuses on:
|
||
|
|
|
||
|
|
1. **Large monolithic files** (>500 lines with multiple responsibilities)
|
||
|
|
2. **Phase 2 additions** (dynamic expansion, adaptive sizing, ACE)
|
||
|
|
3. **Phase 6-2.x bug locations** (active counter fix, header magic SEGV fix)
|
||
|
|
4. **Existing Box structure** (leverage current modularization patterns)
|
||
|
|
|
||
|
|
**Key Finding**: The codebase already has good Box structure in `/core/box/` (40% of code), but **core allocator files remain monolithic**. Breaking these into Boxes would prevent future bugs and accelerate development.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 1. Current Box Structure
|
||
|
|
|
||
|
|
### Existing Boxes (core/box/)
|
||
|
|
|
||
|
|
| File | Lines | Responsibility |
|
||
|
|
|------|-------|----------------|
|
||
|
|
| `hak_core_init.inc.h` | 332 | Initialization & environment parsing |
|
||
|
|
| `pool_core_api.inc.h` | 327 | Pool core allocation API |
|
||
|
|
| `pool_api.inc.h` | 303 | Pool public API |
|
||
|
|
| `pool_mf2_core.inc.h` | 285 | Pool MF2 (Mid-Fast-2) core |
|
||
|
|
| `hak_free_api.inc.h` | 274 | Free API (header dispatch) |
|
||
|
|
| `pool_mf2_types.inc.h` | 266 | Pool MF2 type definitions |
|
||
|
|
| `hak_wrappers.inc.h` | 208 | malloc/free wrappers |
|
||
|
|
| `mailbox_box.c` | 207 | Remote free mailbox |
|
||
|
|
| `hak_alloc_api.inc.h` | 179 | Allocation API |
|
||
|
|
| `pool_init_api.inc.h` | 140 | Pool initialization |
|
||
|
|
| `pool_mf2_helpers.inc.h` | 158 | Pool MF2 helpers |
|
||
|
|
| **+ 13 smaller boxes** | <140 ea | Specialized functions |
|
||
|
|
|
||
|
|
**Total Box coverage**: ~40% of codebase
|
||
|
|
**Unboxed core code**: hakmem_tiny.c (1812), hakmem_tiny_superslab.c (1026), tiny_superslab_alloc.inc.h (749), etc.
|
||
|
|
|
||
|
|
### Box Theory Compliance
|
||
|
|
|
||
|
|
✅ **Good**:
|
||
|
|
- Pool allocator is well-boxed (pool_*.inc.h)
|
||
|
|
- Free path has clear boxes (free_local, free_remote, free_publish)
|
||
|
|
- API boundary is clean (hak_alloc_api, hak_free_api)
|
||
|
|
|
||
|
|
❌ **Missing**:
|
||
|
|
- Tiny allocator core is monolithic (hakmem_tiny.c = 1812 lines)
|
||
|
|
- SuperSlab management has mixed responsibilities (allocation + stats + ACE + caching)
|
||
|
|
- Refill/Adoption logic is intertwined (no clear boundary)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. Large Files Analysis
|
||
|
|
|
||
|
|
### Top 10 Largest Files
|
||
|
|
|
||
|
|
| File | Lines | Responsibilities | Box Potential |
|
||
|
|
|------|-------|-----------------|---------------|
|
||
|
|
| **hakmem_tiny.c** | 1812 | Main allocator, TLS, stats, lifecycle, refill | 🔴 HIGH (5-7 boxes) |
|
||
|
|
| **hakmem_l25_pool.c** | 1195 | L2.5 pool (64KB-1MB) | 🟡 MEDIUM (2-3 boxes) |
|
||
|
|
| **hakmem_tiny_superslab.c** | 1026 | SS alloc, stats, ACE, cache, expansion | 🔴 HIGH (4-5 boxes) |
|
||
|
|
| **hakmem_pool.c** | 907 | L2 pool (1-32KB) | 🟡 MEDIUM (2-3 boxes) |
|
||
|
|
| **hakmem_tiny_stats.c** | 818 | Statistics collection | 🟢 LOW (already focused) |
|
||
|
|
| **tiny_superslab_alloc.inc.h** | 749 | Slab alloc, refill, adoption | 🔴 HIGH (3-4 boxes) |
|
||
|
|
| **tiny_remote.c** | 662 | Remote free handling | 🟡 MEDIUM (2 boxes) |
|
||
|
|
| **hakmem_learner.c** | 603 | Adaptive learning | 🟢 LOW (single responsibility) |
|
||
|
|
| **hakmem_mid_mt.c** | 563 | Mid allocator (multi-thread) | 🟡 MEDIUM (2 boxes) |
|
||
|
|
| **tiny_alloc_fast.inc.h** | 542 | Fast path allocation | 🟡 MEDIUM (2 boxes) |
|
||
|
|
|
||
|
|
**Total**: 9,477 lines in top 10 files (36% of codebase)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. Box Refactoring Candidates
|
||
|
|
|
||
|
|
### 🔴 PRIORITY 1: hakmem_tiny_superslab.c (1026 lines)
|
||
|
|
|
||
|
|
**Current Responsibilities** (5 major):
|
||
|
|
1. **OS-level SuperSlab allocation** (mmap, alignment, munmap) - Lines 187-250
|
||
|
|
2. **Statistics tracking** (global counters, per-class counters) - Lines 22-108
|
||
|
|
3. **Dynamic Expansion** (Phase 2a: chunk management) - Lines 498-650
|
||
|
|
4. **ACE (Adaptive Cache Engine)** (Phase 8.3: promotion/demotion) - Lines 110-1026
|
||
|
|
5. **SuperSlab caching** (precharge, pop, push) - Lines 252-322
|
||
|
|
|
||
|
|
**Proposed Boxes**:
|
||
|
|
|
||
|
|
#### Box: `superslab_os_box.c` (OS Layer)
|
||
|
|
- **Lines**: 187-250, 656-698
|
||
|
|
- **Responsibility**: mmap/munmap, alignment, OS resource management
|
||
|
|
- **Interface**: `superslab_os_acquire()`, `superslab_os_release()`
|
||
|
|
- **Benefit**: Isolate syscall layer (easier to test, mock, port)
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `superslab_stats_box.c` (Statistics)
|
||
|
|
- **Lines**: 22-108, 799-856
|
||
|
|
- **Responsibility**: Global counters, per-class tracking, printing
|
||
|
|
- **Interface**: `ss_stats_*()` functions
|
||
|
|
- **Benefit**: Stats can be disabled/enabled without touching allocation
|
||
|
|
- **Effort**: 1 day
|
||
|
|
|
||
|
|
#### Box: `superslab_expansion_box.c` (Dynamic Expansion)
|
||
|
|
- **Lines**: 498-650
|
||
|
|
- **Responsibility**: SuperSlabHead management, chunk linking, expansion
|
||
|
|
- **Interface**: `init_superslab_head()`, `expand_superslab_head()`, `find_chunk_for_ptr()`
|
||
|
|
- **Benefit**: **Phase 2a code isolation** - all expansion logic in one place
|
||
|
|
- **Bug Prevention**: Active counter bugs (Phase 6-2.3) would be contained here
|
||
|
|
- **Effort**: 3 days
|
||
|
|
|
||
|
|
#### Box: `superslab_ace_box.c` (ACE Engine)
|
||
|
|
- **Lines**: 110-117, 836-1026
|
||
|
|
- **Responsibility**: Adaptive Cache Engine (promotion/demotion, observation)
|
||
|
|
- **Interface**: `hak_tiny_superslab_ace_tick()`, `hak_tiny_superslab_ace_observe_all()`
|
||
|
|
- **Benefit**: **Phase 8.3 isolation** - ACE can be A/B tested independently
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `superslab_cache_box.c` (Cache Management)
|
||
|
|
- **Lines**: 50-322
|
||
|
|
- **Responsibility**: Precharge, pop, push, cache lifecycle
|
||
|
|
- **Interface**: `ss_cache_*()` functions
|
||
|
|
- **Benefit**: Cache layer can be tuned/disabled without affecting allocation
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
**Total Reduction**: 1026 → ~150 lines (core glue code only)
|
||
|
|
**Effort**: 10 days (2 weeks)
|
||
|
|
**Impact**: 🔴🔴🔴 **CRITICAL** - Most bugs occurred here (active counter, OOM, etc.)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🔴 PRIORITY 2: tiny_superslab_alloc.inc.h (749 lines)
|
||
|
|
|
||
|
|
**Current Responsibilities** (3 major):
|
||
|
|
1. **Slab allocation** (linear + freelist modes) - Lines 16-134
|
||
|
|
2. **Refill logic** (adoption, registry scan, expansion integration) - Lines 137-518
|
||
|
|
3. **Main allocation entry point** (hak_tiny_alloc_superslab) - Lines 521-749
|
||
|
|
|
||
|
|
**Proposed Boxes**:
|
||
|
|
|
||
|
|
#### Box: `slab_alloc_box.inc.h` (Slab Allocation)
|
||
|
|
- **Lines**: 16-134
|
||
|
|
- **Responsibility**: Allocate from slab (linear/freelist, remote drain)
|
||
|
|
- **Interface**: `superslab_alloc_from_slab()`
|
||
|
|
- **Benefit**: **Phase 6.24 lazy freelist logic** isolated
|
||
|
|
- **Effort**: 1 day
|
||
|
|
|
||
|
|
#### Box: `slab_refill_box.inc.h` (Refill Logic)
|
||
|
|
- **Lines**: 137-518
|
||
|
|
- **Responsibility**: TLS slab refill (adoption, registry, expansion, mmap)
|
||
|
|
- **Interface**: `superslab_refill()`
|
||
|
|
- **Benefit**: **Complex refill paths** (8 different strategies!) in one testable unit
|
||
|
|
- **Bug Prevention**: Adoption race conditions (Phase 6-2.x) would be easier to debug
|
||
|
|
- **Effort**: 3 days
|
||
|
|
|
||
|
|
#### Box: `slab_fastpath_box.inc.h` (Fast Path)
|
||
|
|
- **Lines**: 521-749
|
||
|
|
- **Responsibility**: Main allocation entry (TLS cache check, fast/slow dispatch)
|
||
|
|
- **Interface**: `hak_tiny_alloc_superslab()`
|
||
|
|
- **Benefit**: Hot path optimization separate from cold path complexity
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
**Total Reduction**: 749 → ~50 lines (header includes only)
|
||
|
|
**Effort**: 6 days (1 week)
|
||
|
|
**Impact**: 🔴🔴 **HIGH** - Refill bugs are common (Phase 6-2.3 active counter fix)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🔴 PRIORITY 3: hakmem_tiny.c (1812 lines)
|
||
|
|
|
||
|
|
**Current State**: Monolithic "God Object"
|
||
|
|
|
||
|
|
**Responsibilities** (7+ major):
|
||
|
|
1. TLS management (g_tls_slabs, g_tls_sll_head, etc.)
|
||
|
|
2. Size class mapping
|
||
|
|
3. Statistics (wrapper counters, path counters)
|
||
|
|
4. Lifecycle (init, shutdown, cleanup)
|
||
|
|
5. Debug/Trace (ring buffer, route tracking)
|
||
|
|
6. Refill orchestration
|
||
|
|
7. Configuration parsing
|
||
|
|
|
||
|
|
**Proposed Boxes** (Top 5):
|
||
|
|
|
||
|
|
#### Box: `tiny_tls_box.c` (TLS Management)
|
||
|
|
- **Responsibility**: TLS variable declarations, initialization, cleanup
|
||
|
|
- **Lines**: ~300
|
||
|
|
- **Interface**: `tiny_tls_init()`, `tiny_tls_get()`, `tiny_tls_cleanup()`
|
||
|
|
- **Benefit**: TLS bugs (Phase 6-2.2 Sanitizer fix) would be isolated
|
||
|
|
- **Effort**: 3 days
|
||
|
|
|
||
|
|
#### Box: `tiny_lifecycle_box.c` (Lifecycle)
|
||
|
|
- **Responsibility**: Constructor/destructor, init, shutdown, cleanup
|
||
|
|
- **Lines**: ~250
|
||
|
|
- **Interface**: `hakmem_tiny_init()`, `hakmem_tiny_shutdown()`, `hakmem_tiny_cleanup()`
|
||
|
|
- **Benefit**: Initialization order bugs easier to debug
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `tiny_config_box.c` (Configuration)
|
||
|
|
- **Responsibility**: Environment variable parsing, config validation
|
||
|
|
- **Lines**: ~200
|
||
|
|
- **Interface**: `tiny_config_parse()`, `tiny_config_get()`
|
||
|
|
- **Benefit**: Config can be unit-tested independently
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `tiny_class_box.c` (Size Classes)
|
||
|
|
- **Responsibility**: Size→class mapping, class sizes, class metadata
|
||
|
|
- **Lines**: ~150
|
||
|
|
- **Interface**: `hak_tiny_size_to_class()`, `hak_tiny_class_size()`
|
||
|
|
- **Benefit**: Class mapping logic isolated (easier to tune/test)
|
||
|
|
- **Effort**: 1 day
|
||
|
|
|
||
|
|
#### Box: `tiny_debug_box.c` (Debug/Trace)
|
||
|
|
- **Responsibility**: Ring buffer, route tracking, failfast, diagnostics
|
||
|
|
- **Lines**: ~300
|
||
|
|
- **Interface**: `tiny_debug_*()` functions
|
||
|
|
- **Benefit**: Debug overhead can be compiled out cleanly
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
**Total Reduction**: 1812 → ~600 lines (core orchestration)
|
||
|
|
**Effort**: 10 days (2 weeks)
|
||
|
|
**Impact**: 🔴🔴🔴 **CRITICAL** - Reduces complexity of main allocator file
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🟡 PRIORITY 4: hakmem_l25_pool.c (1195 lines)
|
||
|
|
|
||
|
|
**Current Responsibilities** (3 major):
|
||
|
|
1. **TLS two-tier cache** (ring + LIFO) - Lines 64-89
|
||
|
|
2. **Global freelist** (sharded, per-class) - Lines 91-100
|
||
|
|
3. **ActiveRun** (bump allocation) - Lines 82-89
|
||
|
|
|
||
|
|
**Proposed Boxes**:
|
||
|
|
|
||
|
|
#### Box: `l25_tls_box.c` (TLS Cache)
|
||
|
|
- **Lines**: ~300
|
||
|
|
- **Responsibility**: TLS ring + LIFO management
|
||
|
|
- **Interface**: `l25_tls_pop()`, `l25_tls_push()`
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `l25_global_box.c` (Global Pool)
|
||
|
|
- **Lines**: ~400
|
||
|
|
- **Responsibility**: Global freelist, sharding, locks
|
||
|
|
- **Interface**: `l25_global_pop()`, `l25_global_push()`
|
||
|
|
- **Effort**: 3 days
|
||
|
|
|
||
|
|
#### Box: `l25_activerun_box.c` (Bump Allocation)
|
||
|
|
- **Lines**: ~200
|
||
|
|
- **Responsibility**: ActiveRun lifecycle, bump pointer
|
||
|
|
- **Interface**: `l25_run_alloc()`, `l25_run_create()`
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
**Total Reduction**: 1195 → ~300 lines (orchestration)
|
||
|
|
**Effort**: 7 days (1 week)
|
||
|
|
**Impact**: 🟡 **MEDIUM** - L2.5 is stable but large
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🟡 PRIORITY 5: tiny_alloc_fast.inc.h (542 lines)
|
||
|
|
|
||
|
|
**Current Responsibilities** (2 major):
|
||
|
|
1. **SFC (Super Front Cache)** - Box 5-NEW integration - Lines 1-200
|
||
|
|
2. **SLL (Single-Linked List)** - Fast path pop - Lines 201-400
|
||
|
|
3. **Profiling/Stats** - RDTSC, counters - Lines 84-152
|
||
|
|
|
||
|
|
**Proposed Boxes**:
|
||
|
|
|
||
|
|
#### Box: `tiny_sfc_box.inc.h` (Super Front Cache)
|
||
|
|
- **Lines**: ~200
|
||
|
|
- **Responsibility**: SFC layer (Layer 0, 128-256 slots)
|
||
|
|
- **Interface**: `sfc_pop()`, `sfc_push()`
|
||
|
|
- **Benefit**: **Box 5-NEW isolation** - SFC can be A/B tested
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
#### Box: `tiny_sll_box.inc.h` (SLL Fast Path)
|
||
|
|
- **Lines**: ~200
|
||
|
|
- **Responsibility**: TLS freelist (Layer 1, unlimited)
|
||
|
|
- **Interface**: `sll_pop()`, `sll_push()`
|
||
|
|
- **Benefit**: Core fast path isolated from SFC complexity
|
||
|
|
- **Effort**: 1 day
|
||
|
|
|
||
|
|
**Total Reduction**: 542 → ~150 lines (orchestration)
|
||
|
|
**Effort**: 3 days
|
||
|
|
**Impact**: 🟡 **MEDIUM** - Fast path is critical but already modular
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🟡 PRIORITY 6: tiny_remote.c (662 lines)
|
||
|
|
|
||
|
|
**Current Responsibilities** (2 major):
|
||
|
|
1. **Remote free tracking** (watch, note, assert) - Lines 1-300
|
||
|
|
2. **Remote queue operations** (MPSC queue) - Lines 301-662
|
||
|
|
|
||
|
|
**Proposed Boxes**:
|
||
|
|
|
||
|
|
#### Box: `remote_track_box.c` (Debug Tracking)
|
||
|
|
- **Lines**: ~300
|
||
|
|
- **Responsibility**: Remote free tracking (debug only)
|
||
|
|
- **Interface**: `tiny_remote_track_*()` functions
|
||
|
|
- **Benefit**: Debug overhead can be compiled out
|
||
|
|
- **Effort**: 1 day
|
||
|
|
|
||
|
|
#### Box: `remote_queue_box.c` (MPSC Queue)
|
||
|
|
- **Lines**: ~362
|
||
|
|
- **Responsibility**: MPSC queue operations (push, pop, drain)
|
||
|
|
- **Interface**: `remote_queue_*()` functions
|
||
|
|
- **Benefit**: Reusable queue component
|
||
|
|
- **Effort**: 2 days
|
||
|
|
|
||
|
|
**Total Reduction**: 662 → ~100 lines (glue)
|
||
|
|
**Effort**: 3 days
|
||
|
|
**Impact**: 🟡 **MEDIUM** - Remote free is stable
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 🟢 PRIORITY 7-10: Smaller Opportunities
|
||
|
|
|
||
|
|
#### 7. `hakmem_pool.c` (907 lines)
|
||
|
|
- **Potential**: Split TLS cache (300 lines) + Global pool (400 lines) + Stats (200 lines)
|
||
|
|
- **Effort**: 5 days
|
||
|
|
- **Impact**: 🟢 LOW - Already stable
|
||
|
|
|
||
|
|
#### 8. `hakmem_mid_mt.c` (563 lines)
|
||
|
|
- **Potential**: Split TLS cache (200 lines) + MT synchronization (200 lines) + Stats (163 lines)
|
||
|
|
- **Effort**: 4 days
|
||
|
|
- **Impact**: 🟢 LOW - Mid allocator works well
|
||
|
|
|
||
|
|
#### 9. `tiny_free_fast.inc.h` (307 lines)
|
||
|
|
- **Potential**: Split ownership check (100 lines) + TLS push (100 lines) + Remote dispatch (107 lines)
|
||
|
|
- **Effort**: 2 days
|
||
|
|
- **Impact**: 🟢 LOW - Already small
|
||
|
|
|
||
|
|
#### 10. `tiny_adaptive_sizing.c` (Phase 2b addition)
|
||
|
|
- **Current**: Already a Box! ✅
|
||
|
|
- **Lines**: ~200 (estimate)
|
||
|
|
- **No action needed** - Good example of Box Theory
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 4. Priority Matrix
|
||
|
|
|
||
|
|
### Effort vs Impact
|
||
|
|
|
||
|
|
```
|
||
|
|
High Impact
|
||
|
|
│
|
||
|
|
│ 1. hakmem_tiny_superslab.c 3. hakmem_tiny.c
|
||
|
|
│ (Boxes: OS, Stats, Expansion, (Boxes: TLS, Lifecycle,
|
||
|
|
│ ACE, Cache) Config, Class, Debug)
|
||
|
|
│ Effort: 10d | Impact: 🔴🔴🔴 Effort: 10d | Impact: 🔴🔴🔴
|
||
|
|
│
|
||
|
|
│ 2. tiny_superslab_alloc.inc.h 4. hakmem_l25_pool.c
|
||
|
|
│ (Boxes: Slab, Refill, Fast) (Boxes: TLS, Global, Run)
|
||
|
|
│ Effort: 6d | Impact: 🔴🔴 Effort: 7d | Impact: 🟡
|
||
|
|
│
|
||
|
|
│ 5. tiny_alloc_fast.inc.h 6. tiny_remote.c
|
||
|
|
│ (Boxes: SFC, SLL) (Boxes: Track, Queue)
|
||
|
|
│ Effort: 3d | Impact: 🟡 Effort: 3d | Impact: 🟡
|
||
|
|
│
|
||
|
|
│ 7-10. Smaller files
|
||
|
|
│ (Various)
|
||
|
|
│ Effort: 2-5d ea | Impact: 🟢
|
||
|
|
│
|
||
|
|
Low Impact
|
||
|
|
└────────────────────────────────────────────────> High Effort
|
||
|
|
1d 3d 5d 7d 10d
|
||
|
|
```
|
||
|
|
|
||
|
|
### Recommended Sequence
|
||
|
|
|
||
|
|
**Phase 1** (Highest ROI):
|
||
|
|
1. **superslab_expansion_box.c** (3 days) - Isolate Phase 2a code
|
||
|
|
2. **superslab_ace_box.c** (2 days) - Isolate Phase 8.3 code
|
||
|
|
3. **slab_refill_box.inc.h** (3 days) - Fix refill complexity
|
||
|
|
|
||
|
|
**Phase 2** (Bug Prevention):
|
||
|
|
4. **tiny_tls_box.c** (3 days) - Prevent TLS bugs
|
||
|
|
5. **tiny_lifecycle_box.c** (2 days) - Prevent init bugs
|
||
|
|
6. **superslab_os_box.c** (2 days) - Isolate syscalls
|
||
|
|
|
||
|
|
**Phase 3** (Long-term Cleanup):
|
||
|
|
7. **superslab_stats_box.c** (1 day)
|
||
|
|
8. **superslab_cache_box.c** (2 days)
|
||
|
|
9. **tiny_config_box.c** (2 days)
|
||
|
|
10. **tiny_class_box.c** (1 day)
|
||
|
|
|
||
|
|
**Total Effort**: ~21 days (4 weeks)
|
||
|
|
**Total Impact**: Reduce top 3 files from 3,587 → ~900 lines (-75%)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 5. Phase 2 & Phase 6-2.x Code Analysis
|
||
|
|
|
||
|
|
### Phase 2a: Dynamic Expansion (hakmem_tiny_superslab.c)
|
||
|
|
|
||
|
|
**Added Code** (Lines 498-650):
|
||
|
|
- `init_superslab_head()` - Initialize per-class chunk list
|
||
|
|
- `expand_superslab_head()` - Allocate new chunk
|
||
|
|
- `find_chunk_for_ptr()` - Locate chunk for pointer
|
||
|
|
|
||
|
|
**Bug History**:
|
||
|
|
- Phase 6-2.3: Active counter bug (lines 575-577) - Missing `ss_active_add()` call
|
||
|
|
- OOM diagnostics (lines 122-185) - Lock depth fix to prevent LIBC malloc
|
||
|
|
|
||
|
|
**Recommendation**: **Extract to `superslab_expansion_box.c`**
|
||
|
|
**Benefit**: All expansion bugs isolated, easier to test/debug
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Phase 2b: Adaptive TLS Cache Sizing
|
||
|
|
|
||
|
|
**Files**:
|
||
|
|
- `tiny_adaptive_sizing.c` - **Already a Box!** ✅
|
||
|
|
- `tiny_adaptive_sizing.h` - Clean interface
|
||
|
|
|
||
|
|
**No action needed** - This is a good example to follow.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Phase 8.3: ACE (Adaptive Cache Engine)
|
||
|
|
|
||
|
|
**Added Code** (hakmem_tiny_superslab.c, Lines 110-117, 836-1026):
|
||
|
|
- `SuperSlabACEState g_ss_ace[]` - Per-class state
|
||
|
|
- `hak_tiny_superslab_ace_tick()` - Promotion/demotion logic
|
||
|
|
- `hak_tiny_superslab_ace_observe_all()` - Registry-based observation
|
||
|
|
|
||
|
|
**Recommendation**: **Extract to `superslab_ace_box.c`**
|
||
|
|
**Benefit**: ACE can be A/B tested, disabled, or replaced independently
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Phase 6-2.x: Bug Locations
|
||
|
|
|
||
|
|
#### Bug #1: Active Counter Double-Decrement (Phase 6-2.3)
|
||
|
|
- **File**: `core/hakmem_tiny_refill_p0.inc.h:103`
|
||
|
|
- **Fix**: Added `ss_active_add(tls->ss, from_freelist);`
|
||
|
|
- **Root Cause**: Refill path didn't increment counter when moving blocks from freelist to TLS
|
||
|
|
- **Box Impact**: If `slab_refill_box.inc.h` existed, bug would be contained in one file
|
||
|
|
|
||
|
|
#### Bug #2: Header Magic SEGV (Phase 6-2.3)
|
||
|
|
- **File**: `core/box/hak_free_api.inc.h:113-131`
|
||
|
|
- **Fix**: Added `hak_is_memory_readable()` check before dereferencing header
|
||
|
|
- **Root Cause**: Registry lookup failure → raw header dispatch → unmapped memory deref
|
||
|
|
- **Box Impact**: Already in a Box! (`hak_free_api.inc.h`) - Good containment
|
||
|
|
|
||
|
|
#### Bug #3: Sanitizer TLS Init (Phase 6-2.2)
|
||
|
|
- **File**: `Makefile:810-828` + `core/tiny_fastcache.c:231-305`
|
||
|
|
- **Fix**: Added `-DHAKMEM_FORCE_LIBC_ALLOC_BUILD=1` to Sanitizer builds
|
||
|
|
- **Root Cause**: ASan `dlsym()` → `malloc()` → TLS uninitialized SEGV
|
||
|
|
- **Box Impact**: If `tiny_tls_box.c` existed, TLS init would be easier to debug
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 6. Implementation Roadmap
|
||
|
|
|
||
|
|
### Week 1-2: SuperSlab Expansion & ACE (Phase 1)
|
||
|
|
|
||
|
|
**Goals**:
|
||
|
|
- Isolate Phase 2a dynamic expansion code
|
||
|
|
- Isolate Phase 8.3 ACE engine
|
||
|
|
- Fix refill complexity
|
||
|
|
|
||
|
|
**Tasks**:
|
||
|
|
1. **Day 1-3**: Create `superslab_expansion_box.c`
|
||
|
|
- Move `init_superslab_head()`, `expand_superslab_head()`, `find_chunk_for_ptr()`
|
||
|
|
- Add unit tests for expansion logic
|
||
|
|
- Verify Phase 6-2.3 active counter fix is contained
|
||
|
|
|
||
|
|
2. **Day 4-5**: Create `superslab_ace_box.c`
|
||
|
|
- Move ACE state, tick, observe functions
|
||
|
|
- Add A/B testing flag (`HAKMEM_ACE_ENABLED=0/1`)
|
||
|
|
- Verify ACE can be disabled without recompile
|
||
|
|
|
||
|
|
3. **Day 6-8**: Create `slab_refill_box.inc.h`
|
||
|
|
- Move `superslab_refill()` (400+ lines!)
|
||
|
|
- Split into sub-functions: adopt, registry_scan, expansion, mmap
|
||
|
|
- Add debug tracing for each refill path
|
||
|
|
|
||
|
|
**Deliverables**:
|
||
|
|
- 3 new Box files
|
||
|
|
- Unit tests for expansion + ACE
|
||
|
|
- Refactoring guide for future Boxes
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Week 3-4: TLS & Lifecycle (Phase 2)
|
||
|
|
|
||
|
|
**Goals**:
|
||
|
|
- Isolate TLS management (prevent Sanitizer bugs)
|
||
|
|
- Isolate lifecycle (prevent init order bugs)
|
||
|
|
- Isolate OS syscalls
|
||
|
|
|
||
|
|
**Tasks**:
|
||
|
|
1. **Day 9-11**: Create `tiny_tls_box.c`
|
||
|
|
- Move TLS variable declarations
|
||
|
|
- Add `tiny_tls_init()`, `tiny_tls_cleanup()`
|
||
|
|
- Fix Sanitizer init order (constructor priority)
|
||
|
|
|
||
|
|
2. **Day 12-13**: Create `tiny_lifecycle_box.c`
|
||
|
|
- Move constructor/destructor
|
||
|
|
- Add `hakmem_tiny_init()`, `hakmem_tiny_shutdown()`
|
||
|
|
- Document init order dependencies
|
||
|
|
|
||
|
|
3. **Day 14-15**: Create `superslab_os_box.c`
|
||
|
|
- Move `superslab_os_acquire()`, `superslab_os_release()`
|
||
|
|
- Add mmap tracing (`HAKMEM_MMAP_TRACE=1`)
|
||
|
|
- Add OOM diagnostics box
|
||
|
|
|
||
|
|
**Deliverables**:
|
||
|
|
- 3 new Box files
|
||
|
|
- Sanitizer builds pass all tests
|
||
|
|
- Init/shutdown documentation
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Week 5-6: Cleanup & Long-term (Phase 3)
|
||
|
|
|
||
|
|
**Goals**:
|
||
|
|
- Finish SuperSlab boxes
|
||
|
|
- Extract config, class, debug boxes
|
||
|
|
- Reduce hakmem_tiny.c to <600 lines
|
||
|
|
|
||
|
|
**Tasks**:
|
||
|
|
1. **Day 16**: Create `superslab_stats_box.c`
|
||
|
|
2. **Day 17-18**: Create `superslab_cache_box.c`
|
||
|
|
3. **Day 19-20**: Create `tiny_config_box.c`
|
||
|
|
4. **Day 21**: Create `tiny_class_box.c`
|
||
|
|
|
||
|
|
**Deliverables**:
|
||
|
|
- 4 new Box files
|
||
|
|
- hakmem_tiny.c reduced to ~600 lines
|
||
|
|
- Documentation update (CLAUDE.md, DOCS_INDEX.md)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 7. Testing Strategy
|
||
|
|
|
||
|
|
### Unit Tests (Per Box)
|
||
|
|
|
||
|
|
Each new Box should have:
|
||
|
|
1. **Interface tests**: Verify all public functions work correctly
|
||
|
|
2. **Boundary tests**: Verify edge cases (OOM, empty state, full state)
|
||
|
|
3. **Mock tests**: Mock dependencies to isolate Box logic
|
||
|
|
|
||
|
|
**Example**: `superslab_expansion_box_test.c`
|
||
|
|
```c
|
||
|
|
// Test expansion logic without OS syscalls
|
||
|
|
void test_expand_superslab_head(void) {
|
||
|
|
SuperSlabHead* head = init_superslab_head(0);
|
||
|
|
assert(head != NULL);
|
||
|
|
assert(head->total_chunks == 1); // Initial chunk
|
||
|
|
|
||
|
|
int result = expand_superslab_head(head);
|
||
|
|
assert(result == 0);
|
||
|
|
assert(head->total_chunks == 2); // Expanded
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Integration Tests (Box Interactions)
|
||
|
|
|
||
|
|
Test how Boxes interact:
|
||
|
|
1. **Refill → Expansion**: When refill exhausts current chunk, expansion creates new chunk
|
||
|
|
2. **ACE → OS**: When ACE promotes to 2MB, OS layer allocates correct size
|
||
|
|
3. **TLS → Lifecycle**: TLS init happens in correct order during startup
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Regression Tests (Bug Prevention)
|
||
|
|
|
||
|
|
For each historical bug, add a regression test:
|
||
|
|
|
||
|
|
**Bug #1: Active Counter** (`test_active_counter_refill.c`)
|
||
|
|
```c
|
||
|
|
// Verify refill increments active counter correctly
|
||
|
|
void test_active_counter_refill(void) {
|
||
|
|
SuperSlab* ss = superslab_allocate(0);
|
||
|
|
uint32_t initial = atomic_load(&ss->total_active_blocks);
|
||
|
|
|
||
|
|
// Refill from freelist
|
||
|
|
slab_refill_from_freelist(ss, 0, 10);
|
||
|
|
|
||
|
|
uint32_t after = atomic_load(&ss->total_active_blocks);
|
||
|
|
assert(after == initial + 10); // MUST increment!
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
**Bug #2: Header Magic SEGV** (`test_free_unmapped_ptr.c`)
|
||
|
|
```c
|
||
|
|
// Verify free doesn't SEGV on unmapped memory
|
||
|
|
void test_free_unmapped_ptr(void) {
|
||
|
|
void* ptr = (void*)0x12345678; // Unmapped address
|
||
|
|
hak_tiny_free(ptr); // Should NOT crash
|
||
|
|
// (Should route to libc_free or ignore safely)
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 8. Success Metrics
|
||
|
|
|
||
|
|
### Code Quality Metrics
|
||
|
|
|
||
|
|
| Metric | Before | After | Improvement |
|
||
|
|
|--------|--------|-------|-------------|
|
||
|
|
| Max file size | 1812 lines | ~600 lines | -67% |
|
||
|
|
| Top 3 file avg | 1196 lines | ~300 lines | -75% |
|
||
|
|
| Avg function size | ~100 lines | ~30 lines | -70% |
|
||
|
|
| Cyclomatic complexity | 200+ (hakmem_tiny.c) | <50 (per Box) | -75% |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Developer Experience Metrics
|
||
|
|
|
||
|
|
| Metric | Before | After | Improvement |
|
||
|
|
|--------|--------|-------|-------------|
|
||
|
|
| Time to find bug location | 30-60 min | 5-10 min | -80% |
|
||
|
|
| Time to add unit test | Hard (monolith) | Easy (per Box) | 5x faster |
|
||
|
|
| Time to A/B test feature | Recompile all | Toggle Box flag | 10x faster |
|
||
|
|
| Onboarding time (new dev) | 2-3 weeks | 1 week | -50% |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Bug Prevention Metrics
|
||
|
|
|
||
|
|
Track bugs by category:
|
||
|
|
|
||
|
|
| Bug Type | Historical Count (Phase 6-7) | Expected After Boxing |
|
||
|
|
|----------|------------------------------|----------------------|
|
||
|
|
| Active counter bugs | 2 | 0 (contained in refill box) |
|
||
|
|
| TLS init bugs | 1 | 0 (contained in tls box) |
|
||
|
|
| OOM diagnostic bugs | 3 | 0 (contained in os box) |
|
||
|
|
| Refill race bugs | 4 | 1-2 (isolated, easier to fix) |
|
||
|
|
|
||
|
|
**Target**: -70% bug count in Phase 8+
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 9. Risks & Mitigation
|
||
|
|
|
||
|
|
### Risk #1: Regression During Refactoring
|
||
|
|
|
||
|
|
**Likelihood**: Medium
|
||
|
|
**Impact**: High (performance regression, new bugs)
|
||
|
|
|
||
|
|
**Mitigation**:
|
||
|
|
1. **Incremental refactoring**: One Box at a time (1 week iterations)
|
||
|
|
2. **A/B testing**: Keep old code with `#ifdef HAKMEM_USE_NEW_BOX`
|
||
|
|
3. **Continuous benchmarking**: Run Larson after each Box
|
||
|
|
4. **Regression tests**: Add test for every moved function
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Risk #2: Performance Overhead from Indirection
|
||
|
|
|
||
|
|
**Likelihood**: Low
|
||
|
|
**Impact**: Medium (-5-10% performance)
|
||
|
|
|
||
|
|
**Mitigation**:
|
||
|
|
1. **Inline hot paths**: Use `static inline` for Box interfaces
|
||
|
|
2. **Link-time optimization**: `-flto` to inline across files
|
||
|
|
3. **Profile-guided optimization**: Use PGO to optimize Box boundaries
|
||
|
|
4. **Benchmark before/after**: Larson, comprehensive, fragmentation stress
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Risk #3: Increased Build Time
|
||
|
|
|
||
|
|
**Likelihood**: Medium
|
||
|
|
**Impact**: Low (few extra seconds)
|
||
|
|
|
||
|
|
**Mitigation**:
|
||
|
|
1. **Parallel make**: Use `make -j8` (already done)
|
||
|
|
2. **Header guards**: Prevent duplicate includes
|
||
|
|
3. **Precompiled headers**: Cache common headers
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 10. Recommendations
|
||
|
|
|
||
|
|
### Immediate Actions (This Week)
|
||
|
|
|
||
|
|
1. ✅ **Review this analysis** with team/user
|
||
|
|
2. ✅ **Pick Phase 1 targets**: superslab_expansion_box, superslab_ace_box, slab_refill_box
|
||
|
|
3. ✅ **Create Box template**: Standard structure (interface, impl, tests)
|
||
|
|
4. ✅ **Set up CI/CD**: Automated tests for each Box
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Short-term (Next 2 Weeks)
|
||
|
|
|
||
|
|
1. **Implement Phase 1 Boxes** (expansion, ACE, refill)
|
||
|
|
2. **Add unit tests** for each Box
|
||
|
|
3. **Run benchmarks** to verify no regression
|
||
|
|
4. **Update documentation** (CLAUDE.md, DOCS_INDEX.md)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Long-term (Next 2 Months)
|
||
|
|
|
||
|
|
1. **Complete all 10 priority Boxes**
|
||
|
|
2. **Reduce hakmem_tiny.c to <600 lines**
|
||
|
|
3. **Achieve -70% bug count in Phase 8+**
|
||
|
|
4. **Onboard new developers faster** (1 week vs 2-3 weeks)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 11. Appendix
|
||
|
|
|
||
|
|
### A. Box Theory Principles (Reminder)
|
||
|
|
|
||
|
|
1. **Single Responsibility**: One Box = One job
|
||
|
|
2. **Clear Boundaries**: Interface is explicit (`.h` file)
|
||
|
|
3. **Testability**: Each Box has unit tests
|
||
|
|
4. **Maintainability**: Code is easy to read, understand, modify
|
||
|
|
5. **A/B Testing**: Boxes can be toggled via flags
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### B. Existing Box Examples (Good Patterns)
|
||
|
|
|
||
|
|
**Good Example #1**: `tiny_adaptive_sizing.c`
|
||
|
|
- **Responsibility**: Adaptive TLS cache sizing (Phase 2b)
|
||
|
|
- **Interface**: `tiny_adaptive_*()` functions in `.h`
|
||
|
|
- **Size**: ~200 lines (focused, testable)
|
||
|
|
- **Dependencies**: Minimal (only TLS state)
|
||
|
|
|
||
|
|
**Good Example #2**: `free_local_box.c`
|
||
|
|
- **Responsibility**: Same-thread freelist push
|
||
|
|
- **Interface**: `free_local_push()`
|
||
|
|
- **Size**: 104 lines (ultra-focused)
|
||
|
|
- **Dependencies**: Only SuperSlab metadata
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### C. Box Template
|
||
|
|
|
||
|
|
```c
|
||
|
|
// ============================================================================
|
||
|
|
// box_name_box.c - One-line description
|
||
|
|
// ============================================================================
|
||
|
|
// Responsibility: What this Box does (1 sentence)
|
||
|
|
// Interface: Public functions (list them)
|
||
|
|
// Dependencies: Other Boxes/modules this depends on
|
||
|
|
// Phase: When this was extracted (e.g., Phase 2a refactoring)
|
||
|
|
//
|
||
|
|
// License: MIT
|
||
|
|
// Date: 2025-11-08
|
||
|
|
|
||
|
|
#include "box_name_box.h"
|
||
|
|
#include "hakmem_internal.h" // Only essential includes
|
||
|
|
|
||
|
|
// ============================================================================
|
||
|
|
// Private Types & Data (Box-local only)
|
||
|
|
// ============================================================================
|
||
|
|
|
||
|
|
typedef struct {
|
||
|
|
// Box-specific state
|
||
|
|
} BoxState;
|
||
|
|
|
||
|
|
static BoxState g_box_state = {0};
|
||
|
|
|
||
|
|
// ============================================================================
|
||
|
|
// Private Functions (static - not exposed)
|
||
|
|
// ============================================================================
|
||
|
|
|
||
|
|
static int box_helper_function(int param) {
|
||
|
|
// Implementation
|
||
|
|
return 0;
|
||
|
|
}
|
||
|
|
|
||
|
|
// ============================================================================
|
||
|
|
// Public Interface (exposed via .h)
|
||
|
|
// ============================================================================
|
||
|
|
|
||
|
|
int box_public_function(int param) {
|
||
|
|
// Implementation
|
||
|
|
return box_helper_function(param);
|
||
|
|
}
|
||
|
|
|
||
|
|
// ============================================================================
|
||
|
|
// Unit Tests (optional - can be separate file)
|
||
|
|
// ============================================================================
|
||
|
|
|
||
|
|
#ifdef HAKMEM_BOX_UNIT_TEST
|
||
|
|
void box_name_test_suite(void) {
|
||
|
|
// Test cases
|
||
|
|
assert(box_public_function(0) == 0);
|
||
|
|
}
|
||
|
|
#endif
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### D. Further Reading
|
||
|
|
|
||
|
|
- **Box Theory**: `/mnt/workdisk/public_share/hakmem/core/box/README.md` (if exists)
|
||
|
|
- **Phase 2a Report**: `/mnt/workdisk/public_share/hakmem/REMAINING_BUGS_ANALYSIS.md`
|
||
|
|
- **Phase 6-2.x Fixes**: `/mnt/workdisk/public_share/hakmem/CLAUDE.md` (lines 45-150)
|
||
|
|
- **Larson Guide**: `/mnt/workdisk/public_share/hakmem/LARSON_GUIDE.md`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
**END OF REPORT**
|
||
|
|
|
||
|
|
Generated by: Claude Task Agent (Ultrathink)
|
||
|
|
Date: 2025-11-08
|
||
|
|
Analysis Time: ~30 minutes
|
||
|
|
Files Analyzed: 50+
|
||
|
|
Recommendations: 10 high-priority Boxes
|
||
|
|
Estimated Effort: 21 days (4 weeks)
|
||
|
|
Expected Impact: -75% code size in top 3 files, -70% bug count
|