tomoaki/hakmem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase E3-FINAL: Fix Box API offset bugs - ALL classes now use correct offsets

Browse Source 
## Root Cause Analysis (GPT5)

**Physical Layout Constraints**:
- Class 0: 8B = [1B header][7B payload] → offset 1 = 9B needed = ❌ IMPOSSIBLE
- Class 1-6: >=16B = [1B header][15B+ payload] → offset 1 = ✅ POSSIBLE
- Class 7: 1KB → offset 0 (compatibility)

**Correct Specification**:
- HAKMEM_TINY_HEADER_CLASSIDX != 0:
  - Class 0, 7: next at offset 0 (overwrites header when on freelist)
  - Class 1-6: next at offset 1 (after header)
- HAKMEM_TINY_HEADER_CLASSIDX == 0:
  - All classes: next at offset 0

**Previous Bug**:
- Attempted "ALL classes offset 1" unification
- Class 0 with offset 1 caused immediate SEGV (9B > 8B block size)
- Mixed 2-arg/3-arg API caused confusion

## Fixes Applied

### 1. Restored 3-Argument Box API (core/box/tiny_next_ptr_box.h)
```c
// Correct signatures
void tiny_next_write(int class_idx, void* base, void* next_value)
void* tiny_next_read(int class_idx, const void* base)

// Correct offset calculation
size_t offset = (class_idx == 0 || class_idx == 7) ? 0 : 1;
```

### 2. Updated 123+ Call Sites Across 34 Files
- hakmem_tiny_hot_pop_v4.inc.h (4 locations)
- hakmem_tiny_fastcache.inc.h (3 locations)
- hakmem_tiny_tls_list.h (12 locations)
- superslab_inline.h (5 locations)
- tiny_fastcache.h (3 locations)
- ptr_trace.h (macro definitions)
- tls_sll_box.h (2 locations)
- + 27 additional files

Pattern: `tiny_next_read(base)` → `tiny_next_read(class_idx, base)`
Pattern: `tiny_next_write(base, next)` → `tiny_next_write(class_idx, base, next)`

### 3. Added Sentinel Detection Guards
- tiny_fast_push(): Block nodes with sentinel in ptr or ptr->next
- tls_list_push(): Block nodes with sentinel in ptr or ptr->next
- Defense-in-depth against remote free sentinel leakage

## Verification (GPT5 Report)

**Test Command**: `./out/release/bench_random_mixed_hakmem --iterations=70000`

**Results**:
- ✅ Main loop completed successfully
- ✅ Drain phase completed successfully
- ✅ NO SEGV (previous crash at iteration 66151 is FIXED)
- ℹ️ Final log: "tiny_alloc(1024) failed" is normal fallback to Mid/ACE layers

**Analysis**:
- Class 0 immediate SEGV: ✅ RESOLVED (correct offset 0 now used)
- 66K iteration crash: ✅ RESOLVED (offset consistency fixed)
- Box API conflicts: ✅ RESOLVED (unified 3-arg API)

## Technical Details

### Offset Logic Justification
```
Class 0:  8B block → next pointer (8B) fits ONLY at offset 0
Class 1: 16B block → next pointer (8B) fits at offset 1 (after 1B header)
Class 2: 32B block → next pointer (8B) fits at offset 1
...
Class 6: 512B block → next pointer (8B) fits at offset 1
Class 7: 1024B block → offset 0 for legacy compatibility
```

### Files Modified (Summary)
- Core API: `box/tiny_next_ptr_box.h`
- Hot paths: `hakmem_tiny_hot_pop*.inc.h`, `tiny_fastcache.h`
- TLS layers: `hakmem_tiny_tls_list.h`, `hakmem_tiny_tls_ops.h`
- SuperSlab: `superslab_inline.h`, `tiny_superslab_*.inc.h`
- Refill: `hakmem_tiny_refill.inc.h`, `tiny_refill_opt.h`
- Free paths: `tiny_free_magazine.inc.h`, `tiny_superslab_free.inc.h`
- Documentation: Multiple Phase E3 reports

## Remaining Work

None for Box API offset bugs - all structural issues resolved.

Future enhancements (non-critical):
- Periodic `grep -R '*(void**)' core/` to detect direct pointer access violations
- Enforce Box API usage via static analysis
- Document offset rationale in architecture docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Moe Charm (CI)
2025-11-13 06:50:20 +09:00
parent bf576e1cb9
commit 72b38bc994
79 changed files with 6865 additions and 1006 deletions
Download Patch File Download Diff File Expand all files Collapse all files

136
core/tiny_refill_opt.h
View File

@ -8,6 +8,7 @@
#include <stdlib.h>
#include "tiny_region_id.h" // For HEADER_MAGIC, HEADER_CLASS_MASK (Fix #6)
#include "ptr_track.h" // Pointer tracking for debugging header corruption
#include "box/tiny_next_ptr_box.h" // Box API: Next pointer read/write
#ifndef HAKMEM_TINY_REFILL_OPT
#define HAKMEM_TINY_REFILL_OPT 1
@ -45,15 +46,10 @@ static inline void refill_opt_dbg(const char* stage, int class_idx, uint32_t n)
// Phase 7 header-aware push_front: link using base+1 for C0-C6 (C7 not used here)
static inline void trc_push_front(TinyRefillChain* c, void* node, int class_idx) {
#if HAKMEM_TINY_HEADER_CLASSIDX
const size_t next_offset = (class_idx == 7) ? 0 : 1;
#else
const size_t next_offset = 0;
#endif
if (c->head == NULL) {
c->head = node; c->tail = node; *(void**)((uint8_t*)node + next_offset) = NULL; c->count = 1;
c->head = node; c->tail = node; tiny_next_write(class_idx, node, NULL); c->count = 1;
} else {
*(void**)((uint8_t*)node + next_offset) = c->head; c->head = node; c->count++;
tiny_next_write(class_idx, node, c->head); c->head = node; c->count++;
}
}
@ -86,7 +82,7 @@ static inline void trc_splice_to_sll(int class_idx, TinyRefillChain* c,
void* cursor = c->head;
uint32_t walked = 0;
while (cursor && walked < c->count + 5) {
void* next = *(void**)((uint8_t*)cursor + 1); // offset 1 for C0
void* next = tiny_next_read(class_idx, cursor);
fprintf(stderr, "[SPLICE_WALK] node=%p next=%p walked=%u/%u\n",
cursor, next, walked, c->count);
if (walked == c->count - 1 && next != NULL) {
@ -100,10 +96,36 @@ static inline void trc_splice_to_sll(int class_idx, TinyRefillChain* c,
fflush(stderr);
}
// 🐛 DEBUG: Log splice call BEFORE calling tls_sll_splice()
#if !HAKMEM_BUILD_RELEASE
{
static _Atomic uint64_t g_splice_call_count = 0;
uint64_t call_num = atomic_fetch_add(&g_splice_call_count, 1);
if (call_num < 10) { // Log first 10 calls
fprintf(stderr, "[TRC_SPLICE #%lu] BEFORE: cls=%d count=%u sll_count_before=%u\n",
call_num, class_idx, c->count, g_tls_sll_count[class_idx]);
fflush(stderr);
}
}
#endif
// CRITICAL: Use Box TLS-SLL API for splice (C7-safe, no race)
// Note: tls_sll_splice() requires capacity parameter (use large value for refill)
uint32_t moved = tls_sll_splice(class_idx, c->head, c->count, 4096);
// 🐛 DEBUG: Log splice result AFTER calling tls_sll_splice()
#if !HAKMEM_BUILD_RELEASE
{
static _Atomic uint64_t g_splice_result_count = 0;
uint64_t result_num = atomic_fetch_add(&g_splice_result_count, 1);
if (result_num < 10) { // Log first 10 results
fprintf(stderr, "[TRC_SPLICE #%lu] AFTER: cls=%d moved=%u/%u sll_count_after=%u\n",
result_num, class_idx, moved, c->count, g_tls_sll_count[class_idx]);
fflush(stderr);
}
}
#endif
// Update sll_count if provided (Box API already updated g_tls_sll_count internally)
// Note: sll_count parameter is typically &g_tls_sll_count[class_idx], already updated
(void)sll_count; // Suppress unused warning
@ -113,6 +135,7 @@ static inline void trc_splice_to_sll(int class_idx, TinyRefillChain* c,
if (__builtin_expect(moved < c->count, 0)) {
fprintf(stderr, "[SPLICE_WARNING] Only moved %u/%u blocks (SLL capacity limit)\n",
moved, c->count);
fflush(stderr);
}
}
@ -183,7 +206,11 @@ static inline uint32_t trc_pop_from_freelist(struct TinySlabMeta* meta,
fprintf(stderr, "[FREELIST_CORRUPT] Head pointer is corrupted (invalid range/alignment)\n");
trc_failfast_abort("freelist_head", class_idx, ss_base, ss_limit, p);
}
void* next = *(void**)p;
// BUG FIX: Use Box API to read next pointer at correct offset
// ROOT CAUSE: Freelist writes next at offset 1 (via tiny_next_write in Box API),
// but this line was reading at offset 0 (direct access *(void**)p).
// This causes 8-byte pointer offset corruption!
void* next = tiny_next_read(class_idx, p);
if (__builtin_expect(trc_refill_guard_enabled() &&
!trc_ptr_is_valid(ss_base, ss_limit, block_size, next),
0)) {
@ -202,30 +229,29 @@ static inline uint32_t trc_pop_from_freelist(struct TinySlabMeta* meta,
}
meta->freelist = next;
// ✅ FIX #11: Restore header BEFORE trc_push_front
// Phase E1-CORRECT: Restore header BEFORE trc_push_front
// ROOT CAUSE: Freelist stores next at base (offset 0), overwriting header.
// trc_push_front() uses offset=1 for C0-C6, expecting header at base.
// trc_push_front() uses offset=1 for ALL classes, expecting header at base.
// Without restoration, offset=1 contains garbage → chain corruption → SEGV!
//
// SOLUTION: Restore header AFTER reading freelist next, BEFORE chain push.
// Cost: 1 byte write per freelist block (~1-2 cycles, negligible).
// ALL classes (C0-C7) need header restoration!
#if HAKMEM_TINY_HEADER_CLASSIDX
if (class_idx != 7) {
// DEBUG: Log header restoration for class 2
uint8_t before = *(uint8_t*)p;
PTR_TRACK_FREELIST_POP(p, class_idx);
*(uint8_t*)p = HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK);
PTR_TRACK_HEADER_WRITE(p, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
static _Atomic uint64_t g_freelist_count_c2 = 0;
if (class_idx == 2) {
uint64_t fl_num = atomic_fetch_add(&g_freelist_count_c2, 1);
if (fl_num < 100) { // Log first 100 freelist pops
extern _Atomic uint64_t malloc_count;
uint64_t call_num = atomic_load(&malloc_count);
fprintf(stderr, "[FREELIST_HEADER_RESTORE] fl#%lu call=%lu cls=%d ptr=%p before=0x%02x after=0x%02x\n",
fl_num, call_num, class_idx, p, before, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
fflush(stderr);
}
// DEBUG: Log header restoration for class 2
uint8_t before = *(uint8_t*)p;
PTR_TRACK_FREELIST_POP(p, class_idx);
*(uint8_t*)p = HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK);
PTR_TRACK_HEADER_WRITE(p, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
static _Atomic uint64_t g_freelist_count_c2 = 0;
if (class_idx == 2) {
uint64_t fl_num = atomic_fetch_add(&g_freelist_count_c2, 1);
if (fl_num < 100) { // Log first 100 freelist pops
extern _Atomic uint64_t malloc_count;
uint64_t call_num = atomic_load(&malloc_count);
fprintf(stderr, "[FREELIST_HEADER_RESTORE] fl#%lu call=%lu cls=%d ptr=%p before=0x%02x after=0x%02x\n",
fl_num, call_num, class_idx, p, before, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
fflush(stderr);
}
}
#endif
@ -272,30 +298,29 @@ static inline uint32_t trc_linear_carve(uint8_t* base, size_t bs,
(void*)base, meta->carved, batch, (void*)cursor);
}
// ✅ FIX #6: Write headers to carved blocks BEFORE linking
// Phase E1-CORRECT: Write headers to carved blocks BEFORE linking
// ALL classes (C0-C7) have 1-byte headers now
// ROOT CAUSE: tls_sll_splice() checks byte 0 for header magic to determine
// next_offset. Without headers, it finds 0x00 and uses next_offset=0 (WRONG!),
// reading garbage pointers from wrong offset, causing SEGV.
// SOLUTION: Write headers to all carved blocks so splice detection works correctly.
// SOLUTION: Write headers to ALL carved blocks (including C7) so splice detection works correctly.
#if HAKMEM_TINY_HEADER_CLASSIDX
if (class_idx != 7) {
// Write headers to all batch blocks (C0-C6 only, C7 is headerless)
static _Atomic uint64_t g_carve_count = 0;
for (uint32_t i = 0; i < batch; i++) {
uint8_t* block = cursor + (i * stride);
PTR_TRACK_CARVE((void*)block, class_idx);
*block = HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK);
PTR_TRACK_HEADER_WRITE((void*)block, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
// Write headers to all batch blocks (ALL classes C0-C7)
static _Atomic uint64_t g_carve_count = 0;
for (uint32_t i = 0; i < batch; i++) {
uint8_t* block = cursor + (i * stride);
PTR_TRACK_CARVE((void*)block, class_idx);
*block = HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK);
PTR_TRACK_HEADER_WRITE((void*)block, HEADER_MAGIC | (class_idx & HEADER_CLASS_MASK));
// ✅ Option C: Class 2 inline logs - CARVE operation
if (class_idx == 2) {
uint64_t carve_id = atomic_fetch_add(&g_carve_count, 1);
extern _Atomic uint64_t malloc_count;
uint64_t call = atomic_load(&malloc_count);
fprintf(stderr, "[C2_CARVE] ptr=%p header=0xa2 batch_idx=%u/%u carve_id=%lu call=%lu\n",
(void*)block, i+1, batch, carve_id, call);
fflush(stderr);
}
// ✅ Option C: Class 2 inline logs - CARVE operation
if (class_idx == 2) {
uint64_t carve_id = atomic_fetch_add(&g_carve_count, 1);
extern _Atomic uint64_t malloc_count;
uint64_t call = atomic_load(&malloc_count);
fprintf(stderr, "[C2_CARVE] ptr=%p header=0xa2 batch_idx=%u/%u carve_id=%lu call=%lu\n",
(void*)block, i+1, batch, carve_id, call);
fflush(stderr);
}
}
#endif
@ -304,14 +329,9 @@ static inline uint32_t trc_linear_carve(uint8_t* base, size_t bs,
// For header classes (C0-C6), the first byte at base is the 1-byte header.
// Store the SLL next pointer at base+1 to avoid clobbering the header.
// For C7 (headerless), store at base.
#if HAKMEM_TINY_HEADER_CLASSIDX
const size_t next_offset = (class_idx == 7) ? 0 : 1;
#else
const size_t next_offset = 0;
#endif
for (uint32_t i = 1; i < batch; i++) {
uint8_t* next = cursor + stride;
*(void**)(cursor + next_offset) = (void*)next;
tiny_next_write(class_idx, (void*)cursor, (void*)next);
cursor = next;
}
void* tail = (void*)cursor;
@ -321,17 +341,17 @@ static inline uint32_t trc_linear_carve(uint8_t* base, size_t bs,
// allocation, causing SEGV when TLS SLL is traversed (crash at iteration 38,985).
// The loop above only links blocks 0→1, 1→2, ..., (batch-2)→(batch-1).
// It does NOT write to tail's next pointer, leaving stale data!
*(void**)((uint8_t*)tail + next_offset) = NULL;
tiny_next_write(class_idx, tail, NULL);
// Debug: validate first link
#if !HAKMEM_BUILD_RELEASE
if (batch >= 2) {
void* first_next = *(void**)((uint8_t*)head + next_offset);
fprintf(stderr, "[LINEAR_LINK] cls=%d head=%p off=%zu next=%p tail=%p\n",
class_idx, head, (size_t)next_offset, first_next, tail);
void* first_next = tiny_next_read(class_idx, head);
fprintf(stderr, "[LINEAR_LINK] cls=%d head=%p next=%p tail=%p\n",
class_idx, head, first_next, tail);
} else {
fprintf(stderr, "[LINEAR_LINK] cls=%d head=%p off=%zu next=%p tail=%p\n",
class_idx, head, (size_t)next_offset, (void*)0, tail);
fprintf(stderr, "[LINEAR_LINK] cls=%d head=%p next=%p tail=%p\n",
class_idx, head, (void*)0, tail);
}
#endif
// FIX: Update both carved (monotonic) and used (active count)