Files
hakmem/core/hakmem_tiny_superslab_constants.h

84 lines
3.2 KiB
C
Raw Normal View History

Phase 6-2.5: Fix SuperSlab alignment bug + refactor constants ## Problem: 53-byte misalignment mystery **Symptom:** All SuperSlab allocations misaligned by exactly 53 bytes ``` [TRC_FAILFAST_PTR] stage=alloc_ret_align cls=7 ptr=0x..f835 offset=63541 (expected: 63488) Diff: 63541 - 63488 = 53 bytes ``` ## Root Cause (Ultrathink investigation) **sizeof(SuperSlab) != hardcoded offset:** - `sizeof(SuperSlab)` = 1088 bytes (actual struct size) - `tiny_slab_base_for()` used: 1024 (hardcoded) - `superslab_init_slab()` assumed: 2048 (in capacity calc) **Impact:** 1. Memory corruption: 64-byte overlap with SuperSlab metadata 2. Misalignment: 1088 % 1024 = 64 (violates class 7 alignment) 3. Inconsistency: Init assumed 2048, but runtime used 1024 ## Solution ### 1. Centralize constants (NEW) **File:** `core/hakmem_tiny_superslab_constants.h` - `SLAB_SIZE` = 64KB - `SUPERSLAB_HEADER_SIZE` = 1088 - `SUPERSLAB_SLAB0_DATA_OFFSET` = 2048 (aligned to 1024) - `SUPERSLAB_SLAB0_USABLE_SIZE` = 63488 (64KB - 2048) - Compile-time validation checks **Why 2048?** - Round up 1088 to next 1024-byte boundary - Ensures proper alignment for class 7 (1024-byte blocks) - Previous: (1088 + 1023) & ~1023 = 2048 ### 2. Update all code to use constants - `hakmem_tiny_superslab.h`: `tiny_slab_base_for()` → use `SUPERSLAB_SLAB0_DATA_OFFSET` - `hakmem_tiny_superslab.c`: `superslab_init_slab()` → use `SUPERSLAB_SLAB0_USABLE_SIZE` - Removed hardcoded 1024, 2048 magic numbers ### 3. Add class consistency check **File:** `core/tiny_superslab_alloc.inc.h:433-449` - Verify `tls->ss->size_class == class_idx` before allocation - Unbind TLS if mismatch detected - Prevents using wrong block_size for calculations ## Status ⚠️ **INCOMPLETE - New issue discovered** After fix, benchmark hits different error: ``` [TRC_FAILFAST] stage=freelist_next cls=7 node=0x...d474 ``` Freelist corruption detected. Likely caused by: - 2048 offset change affects free() path - Block addresses no longer match freelist expectations - Needs further investigation ## Files Modified - `core/hakmem_tiny_superslab_constants.h` - NEW: Centralized constants - `core/hakmem_tiny_superslab.h` - Use SUPERSLAB_SLAB0_DATA_OFFSET - `core/hakmem_tiny_superslab.c` - Use SUPERSLAB_SLAB0_USABLE_SIZE - `core/tiny_superslab_alloc.inc.h` - Add class consistency check - `core/hakmem_tiny_init.inc` - Remove diet mode override (Phase 6-2.5) - `core/hakmem_super_registry.h` - Remove debug output (cleaned) - `PERFORMANCE_INVESTIGATION_REPORT.md` - Task agent analysis ## Next Steps 1. Investigate freelist corruption with 2048 offset 2. Verify free() path uses tiny_slab_base_for() correctly 3. Consider reverting to 1024 and fixing capacity calculation instead 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-07 21:45:20 +09:00
// hakmem_tiny_superslab_constants.h - SuperSlab Layout Constants
// Purpose: Centralize all SuperSlab layout magic numbers
// Phase 6-2.5: Created to fix sizeof(SuperSlab) vs hardcoded offset mismatch
#ifndef HAKMEM_TINY_SUPERSLAB_CONSTANTS_H
#define HAKMEM_TINY_SUPERSLAB_CONSTANTS_H
// ============================================================================
// SuperSlab Layout Constants
// ============================================================================
// Log2 range for SuperSlab sizes:
// - MIN: 512KB (2^19) - Phase 2 optimization: reduced from 1MB
// - MAX: 2MB (2^21) - unchanged
// - DEFAULT: 512KB (2^19) - Phase 2 optimization: reduced from 2MB
//
// Phase 2-Opt2: Reduce SuperSlab size to minimize initialization cost
// Benefit: 75% reduction in allocation size (2MB → 512KB)
// Expected: +3-5% throughput improvement
// Rationale:
// - Smaller SuperSlab = fewer page faults during allocation
// - Better memory granularity (less wasted VA space)
// - Memset already removed in Phase 1, so pure allocation overhead
// - Perf analysis showed shared_pool_acquire_slab at 23.83% CPU time
#ifndef SUPERSLAB_LG_MIN
#define SUPERSLAB_LG_MIN 19
#endif
#ifndef SUPERSLAB_LG_MAX
#define SUPERSLAB_LG_MAX 21
#endif
#ifndef SUPERSLAB_LG_DEFAULT
#define SUPERSLAB_LG_DEFAULT 19
#endif
Phase 6-2.5: Fix SuperSlab alignment bug + refactor constants ## Problem: 53-byte misalignment mystery **Symptom:** All SuperSlab allocations misaligned by exactly 53 bytes ``` [TRC_FAILFAST_PTR] stage=alloc_ret_align cls=7 ptr=0x..f835 offset=63541 (expected: 63488) Diff: 63541 - 63488 = 53 bytes ``` ## Root Cause (Ultrathink investigation) **sizeof(SuperSlab) != hardcoded offset:** - `sizeof(SuperSlab)` = 1088 bytes (actual struct size) - `tiny_slab_base_for()` used: 1024 (hardcoded) - `superslab_init_slab()` assumed: 2048 (in capacity calc) **Impact:** 1. Memory corruption: 64-byte overlap with SuperSlab metadata 2. Misalignment: 1088 % 1024 = 64 (violates class 7 alignment) 3. Inconsistency: Init assumed 2048, but runtime used 1024 ## Solution ### 1. Centralize constants (NEW) **File:** `core/hakmem_tiny_superslab_constants.h` - `SLAB_SIZE` = 64KB - `SUPERSLAB_HEADER_SIZE` = 1088 - `SUPERSLAB_SLAB0_DATA_OFFSET` = 2048 (aligned to 1024) - `SUPERSLAB_SLAB0_USABLE_SIZE` = 63488 (64KB - 2048) - Compile-time validation checks **Why 2048?** - Round up 1088 to next 1024-byte boundary - Ensures proper alignment for class 7 (1024-byte blocks) - Previous: (1088 + 1023) & ~1023 = 2048 ### 2. Update all code to use constants - `hakmem_tiny_superslab.h`: `tiny_slab_base_for()` → use `SUPERSLAB_SLAB0_DATA_OFFSET` - `hakmem_tiny_superslab.c`: `superslab_init_slab()` → use `SUPERSLAB_SLAB0_USABLE_SIZE` - Removed hardcoded 1024, 2048 magic numbers ### 3. Add class consistency check **File:** `core/tiny_superslab_alloc.inc.h:433-449` - Verify `tls->ss->size_class == class_idx` before allocation - Unbind TLS if mismatch detected - Prevents using wrong block_size for calculations ## Status ⚠️ **INCOMPLETE - New issue discovered** After fix, benchmark hits different error: ``` [TRC_FAILFAST] stage=freelist_next cls=7 node=0x...d474 ``` Freelist corruption detected. Likely caused by: - 2048 offset change affects free() path - Block addresses no longer match freelist expectations - Needs further investigation ## Files Modified - `core/hakmem_tiny_superslab_constants.h` - NEW: Centralized constants - `core/hakmem_tiny_superslab.h` - Use SUPERSLAB_SLAB0_DATA_OFFSET - `core/hakmem_tiny_superslab.c` - Use SUPERSLAB_SLAB0_USABLE_SIZE - `core/tiny_superslab_alloc.inc.h` - Add class consistency check - `core/hakmem_tiny_init.inc` - Remove diet mode override (Phase 6-2.5) - `core/hakmem_super_registry.h` - Remove debug output (cleaned) - `PERFORMANCE_INVESTIGATION_REPORT.md` - Task agent analysis ## Next Steps 1. Investigate freelist corruption with 2048 offset 2. Verify free() path uses tiny_slab_base_for() correctly 3. Consider reverting to 1024 and fixing capacity calculation instead 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-07 21:45:20 +09:00
// Size of each slab within SuperSlab (fixed, never changes)
#define SLAB_SIZE (64 * 1024) // 64KB per slab
Tiny Pool redesign: P0.1, P0.3, P1.1, P1.2 - Out-of-band class_idx lookup This commit implements the first phase of Tiny Pool redesign based on ChatGPT architecture review. The goal is to eliminate Header/Next pointer conflicts by moving class_idx lookup out-of-band (to SuperSlab metadata). ## P0.1: C0(8B) class upgraded to 16B - Size table changed: {16,32,64,128,256,512,1024,2048} (8 classes) - LUT updated: 1..16 → class 0, 17..32 → class 1, etc. - tiny_next_off: C0 now uses offset 1 (header preserved) - Eliminates edge cases for 8B allocations ## P0.3: Slab reuse guard Box (tls_slab_reuse_guard_box.h) - New Box for draining TLS SLL before slab reuse - ENV gate: HAKMEM_TINY_SLAB_REUSE_GUARD=1 - Prevents stale pointers when slabs are recycled - Follows Box theory: single responsibility, minimal API ## P1.1: SuperSlab class_map addition - Added uint8_t class_map[SLABS_PER_SUPERSLAB_MAX] to SuperSlab - Maps slab_idx → class_idx for out-of-band lookup - Initialized to 255 (UNASSIGNED) on SuperSlab creation - Set correctly on slab initialization in all backends ## P1.2: Free fast path uses class_map - ENV gate: HAKMEM_TINY_USE_CLASS_MAP=1 - Free path can now get class_idx from class_map instead of Header - Falls back to Header read if class_map returns invalid value - Fixed Legacy Backend dynamic slab initialization bug ## Documentation added - HAKMEM_ARCHITECTURE_OVERVIEW.md: 4-layer architecture analysis - TLS_SLL_ARCHITECTURE_INVESTIGATION.md: Root cause analysis - PTR_LIFECYCLE_TRACE_AND_ROOT_CAUSE_ANALYSIS.md: Pointer tracking - TINY_REDESIGN_CHECKLIST.md: Implementation roadmap (P0-P3) ## Test results - Baseline: 70% success rate (30% crash - pre-existing issue) - class_map enabled: 70% success rate (same as baseline) - Performance: ~30.5M ops/s (unchanged) ## Next steps (P1.3, P2, P3) - P1.3: Add meta->active for accurate TLS/freelist sync - P2: TLS SLL redesign with Box-based counting - P3: Complete Header out-of-band migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-28 13:42:39 +09:00
// SuperSlab struct size (as of P1.1)
// Actual value: sizeof(SuperSlab) = 1192 bytes
Phase 6-2.5: Fix SuperSlab alignment bug + refactor constants ## Problem: 53-byte misalignment mystery **Symptom:** All SuperSlab allocations misaligned by exactly 53 bytes ``` [TRC_FAILFAST_PTR] stage=alloc_ret_align cls=7 ptr=0x..f835 offset=63541 (expected: 63488) Diff: 63541 - 63488 = 53 bytes ``` ## Root Cause (Ultrathink investigation) **sizeof(SuperSlab) != hardcoded offset:** - `sizeof(SuperSlab)` = 1088 bytes (actual struct size) - `tiny_slab_base_for()` used: 1024 (hardcoded) - `superslab_init_slab()` assumed: 2048 (in capacity calc) **Impact:** 1. Memory corruption: 64-byte overlap with SuperSlab metadata 2. Misalignment: 1088 % 1024 = 64 (violates class 7 alignment) 3. Inconsistency: Init assumed 2048, but runtime used 1024 ## Solution ### 1. Centralize constants (NEW) **File:** `core/hakmem_tiny_superslab_constants.h` - `SLAB_SIZE` = 64KB - `SUPERSLAB_HEADER_SIZE` = 1088 - `SUPERSLAB_SLAB0_DATA_OFFSET` = 2048 (aligned to 1024) - `SUPERSLAB_SLAB0_USABLE_SIZE` = 63488 (64KB - 2048) - Compile-time validation checks **Why 2048?** - Round up 1088 to next 1024-byte boundary - Ensures proper alignment for class 7 (1024-byte blocks) - Previous: (1088 + 1023) & ~1023 = 2048 ### 2. Update all code to use constants - `hakmem_tiny_superslab.h`: `tiny_slab_base_for()` → use `SUPERSLAB_SLAB0_DATA_OFFSET` - `hakmem_tiny_superslab.c`: `superslab_init_slab()` → use `SUPERSLAB_SLAB0_USABLE_SIZE` - Removed hardcoded 1024, 2048 magic numbers ### 3. Add class consistency check **File:** `core/tiny_superslab_alloc.inc.h:433-449` - Verify `tls->ss->size_class == class_idx` before allocation - Unbind TLS if mismatch detected - Prevents using wrong block_size for calculations ## Status ⚠️ **INCOMPLETE - New issue discovered** After fix, benchmark hits different error: ``` [TRC_FAILFAST] stage=freelist_next cls=7 node=0x...d474 ``` Freelist corruption detected. Likely caused by: - 2048 offset change affects free() path - Block addresses no longer match freelist expectations - Needs further investigation ## Files Modified - `core/hakmem_tiny_superslab_constants.h` - NEW: Centralized constants - `core/hakmem_tiny_superslab.h` - Use SUPERSLAB_SLAB0_DATA_OFFSET - `core/hakmem_tiny_superslab.c` - Use SUPERSLAB_SLAB0_USABLE_SIZE - `core/tiny_superslab_alloc.inc.h` - Add class consistency check - `core/hakmem_tiny_init.inc` - Remove diet mode override (Phase 6-2.5) - `core/hakmem_super_registry.h` - Remove debug output (cleaned) - `PERFORMANCE_INVESTIGATION_REPORT.md` - Task agent analysis ## Next Steps 1. Investigate freelist corruption with 2048 offset 2. Verify free() path uses tiny_slab_base_for() correctly 3. Consider reverting to 1024 and fixing capacity calculation instead 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-07 21:45:20 +09:00
// This includes: magic, lg_size, size_class, total_active_blocks,
Tiny Pool redesign: P0.1, P0.3, P1.1, P1.2 - Out-of-band class_idx lookup This commit implements the first phase of Tiny Pool redesign based on ChatGPT architecture review. The goal is to eliminate Header/Next pointer conflicts by moving class_idx lookup out-of-band (to SuperSlab metadata). ## P0.1: C0(8B) class upgraded to 16B - Size table changed: {16,32,64,128,256,512,1024,2048} (8 classes) - LUT updated: 1..16 → class 0, 17..32 → class 1, etc. - tiny_next_off: C0 now uses offset 1 (header preserved) - Eliminates edge cases for 8B allocations ## P0.3: Slab reuse guard Box (tls_slab_reuse_guard_box.h) - New Box for draining TLS SLL before slab reuse - ENV gate: HAKMEM_TINY_SLAB_REUSE_GUARD=1 - Prevents stale pointers when slabs are recycled - Follows Box theory: single responsibility, minimal API ## P1.1: SuperSlab class_map addition - Added uint8_t class_map[SLABS_PER_SUPERSLAB_MAX] to SuperSlab - Maps slab_idx → class_idx for out-of-band lookup - Initialized to 255 (UNASSIGNED) on SuperSlab creation - Set correctly on slab initialization in all backends ## P1.2: Free fast path uses class_map - ENV gate: HAKMEM_TINY_USE_CLASS_MAP=1 - Free path can now get class_idx from class_map instead of Header - Falls back to Header read if class_map returns invalid value - Fixed Legacy Backend dynamic slab initialization bug ## Documentation added - HAKMEM_ARCHITECTURE_OVERVIEW.md: 4-layer architecture analysis - TLS_SLL_ARCHITECTURE_INVESTIGATION.md: Root cause analysis - PTR_LIFECYCLE_TRACE_AND_ROOT_CAUSE_ANALYSIS.md: Pointer tracking - TINY_REDESIGN_CHECKLIST.md: Implementation roadmap (P0-P3) ## Test results - Baseline: 70% success rate (30% crash - pre-existing issue) - class_map enabled: 70% success rate (same as baseline) - Performance: ~30.5M ops/s (unchanged) ## Next steps (P1.3, P2, P3) - P1.3: Add meta->active for accurate TLS/freelist sync - P2: TLS SLL redesign with Box-based counting - P3: Complete Header out-of-band migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-28 13:42:39 +09:00
// remote_heads[], slabs[], slab_listed[], class_map[], etc.
// P1.1: Added class_map[32] (+32 bytes) for out-of-band class_idx lookup
#define SUPERSLAB_HEADER_SIZE 1192
Phase 6-2.5: Fix SuperSlab alignment bug + refactor constants ## Problem: 53-byte misalignment mystery **Symptom:** All SuperSlab allocations misaligned by exactly 53 bytes ``` [TRC_FAILFAST_PTR] stage=alloc_ret_align cls=7 ptr=0x..f835 offset=63541 (expected: 63488) Diff: 63541 - 63488 = 53 bytes ``` ## Root Cause (Ultrathink investigation) **sizeof(SuperSlab) != hardcoded offset:** - `sizeof(SuperSlab)` = 1088 bytes (actual struct size) - `tiny_slab_base_for()` used: 1024 (hardcoded) - `superslab_init_slab()` assumed: 2048 (in capacity calc) **Impact:** 1. Memory corruption: 64-byte overlap with SuperSlab metadata 2. Misalignment: 1088 % 1024 = 64 (violates class 7 alignment) 3. Inconsistency: Init assumed 2048, but runtime used 1024 ## Solution ### 1. Centralize constants (NEW) **File:** `core/hakmem_tiny_superslab_constants.h` - `SLAB_SIZE` = 64KB - `SUPERSLAB_HEADER_SIZE` = 1088 - `SUPERSLAB_SLAB0_DATA_OFFSET` = 2048 (aligned to 1024) - `SUPERSLAB_SLAB0_USABLE_SIZE` = 63488 (64KB - 2048) - Compile-time validation checks **Why 2048?** - Round up 1088 to next 1024-byte boundary - Ensures proper alignment for class 7 (1024-byte blocks) - Previous: (1088 + 1023) & ~1023 = 2048 ### 2. Update all code to use constants - `hakmem_tiny_superslab.h`: `tiny_slab_base_for()` → use `SUPERSLAB_SLAB0_DATA_OFFSET` - `hakmem_tiny_superslab.c`: `superslab_init_slab()` → use `SUPERSLAB_SLAB0_USABLE_SIZE` - Removed hardcoded 1024, 2048 magic numbers ### 3. Add class consistency check **File:** `core/tiny_superslab_alloc.inc.h:433-449` - Verify `tls->ss->size_class == class_idx` before allocation - Unbind TLS if mismatch detected - Prevents using wrong block_size for calculations ## Status ⚠️ **INCOMPLETE - New issue discovered** After fix, benchmark hits different error: ``` [TRC_FAILFAST] stage=freelist_next cls=7 node=0x...d474 ``` Freelist corruption detected. Likely caused by: - 2048 offset change affects free() path - Block addresses no longer match freelist expectations - Needs further investigation ## Files Modified - `core/hakmem_tiny_superslab_constants.h` - NEW: Centralized constants - `core/hakmem_tiny_superslab.h` - Use SUPERSLAB_SLAB0_DATA_OFFSET - `core/hakmem_tiny_superslab.c` - Use SUPERSLAB_SLAB0_USABLE_SIZE - `core/tiny_superslab_alloc.inc.h` - Add class consistency check - `core/hakmem_tiny_init.inc` - Remove diet mode override (Phase 6-2.5) - `core/hakmem_super_registry.h` - Remove debug output (cleaned) - `PERFORMANCE_INVESTIGATION_REPORT.md` - Task agent analysis ## Next Steps 1. Investigate freelist corruption with 2048 offset 2. Verify free() path uses tiny_slab_base_for() correctly 3. Consider reverting to 1024 and fixing capacity calculation instead 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-07 21:45:20 +09:00
// Slab 0 data offset (CRITICAL: Must be aligned to largest block size)
// Phase 6-2.5 FIX: Changed from 1024 to 2048
//
// Why 2048?
// - sizeof(SuperSlab) = 1088 bytes
// - Largest block size = 1024 bytes (class 7)
// - Must round up to next 1024-byte boundary: (1088 + 1023) & ~1023 = 2048
//
// Layout:
// [0..1087] SuperSlab header (1088 bytes)
// [1088..2047] Padding (960 bytes, unused)
// [2048..65535] Slab 0 data (63488 bytes = 64KB - 2048)
//
// Previous value (1024) caused:
// - 64-byte overlap with SuperSlab metadata (corruption)
// - Misalignment for class 7 allocations (1024 % 1024 != 0)
#define SUPERSLAB_SLAB0_DATA_OFFSET 2048
// Slab 0 usable size (for capacity calculation)
#define SUPERSLAB_SLAB0_USABLE_SIZE (SLAB_SIZE - SUPERSLAB_SLAB0_DATA_OFFSET) // 63488 bytes
// Regular slab (i > 0) usable size
#define SUPERSLAB_SLAB_USABLE_SIZE SLAB_SIZE // 65536 bytes
// ============================================================================
// Validation (compile-time check)
// ============================================================================
// Ensure SLAB0_DATA_OFFSET is aligned to largest block size (1024)
#if (SUPERSLAB_SLAB0_DATA_OFFSET % 1024) != 0
#error "SUPERSLAB_SLAB0_DATA_OFFSET must be 1024-byte aligned for class 7"
#endif
// Ensure SLAB0_DATA_OFFSET is large enough to contain SuperSlab header
#if SUPERSLAB_SLAB0_DATA_OFFSET < SUPERSLAB_HEADER_SIZE
#error "SUPERSLAB_SLAB0_DATA_OFFSET must be >= sizeof(SuperSlab)"
#endif
#endif // HAKMEM_TINY_SUPERSLAB_CONSTANTS_H