hakmem/docs/analysis/SEGFAULT_INVESTIGATION_REPORT.md

SEGFAULT Investigation Report - bench_random_mixed & bench_mid_large_mt

Date: 2025-11-07 Status: ✅ ROOT CAUSE IDENTIFIED Priority: CRITICAL

---

Executive Summary

Problem: bench_random_mixed_hakmem and bench_mid_large_mt_hakmem crash with SEGV (exit 139) when direct-linked, but work fine with LD_PRELOAD.

Root Cause: SuperSlab registry lookup failures cause headerless tiny allocations to be misidentified as having HAKMEM headers during free(), leading to:

1. Invalid memory reads at ptr - HEADER_SIZE → SEGV
2. Memory leaks when g_invalid_free_mode=1 skips frees
3. Eventual memory exhaustion or corruption

Why LD_PRELOAD Works: LD_PRELOAD defaults to g_invalid_free_mode=0 (fallback to libc), which masks the issue by routing failed frees to __libc_free().

Why Direct-Link Crashes: Direct-link defaults to g_invalid_free_mode=1 (skip invalid frees), which silently leaks memory until exhaustion.

---

Reproduction

Crashes (Direct-Link)

./bench_random_mixed_hakmem 50000 2048 123
# → Segmentation fault (exit 139)

./bench_mid_large_mt_hakmem 4 40000 2048 42
# → Segmentation fault (exit 139)

Error Output:

[hakmem] ERROR: Invalid magic 0x0 (expected 0x48414B4D)
[hakmem] ERROR: Invalid magic 0x0 (expected 0x48414B4D)
... (hundreds of errors)
free(): invalid pointer
Segmentation fault (core dumped)

Works Fine (LD_PRELOAD)

LD_PRELOAD=./libhakmem_asan.so ./bench_random_mixed_system 200000 4096 1234567
# → 5.7M ops/s ✅

Crash Threshold

• Small workloads: ≤20K ops with 512 slots → Works
• Large workloads: ≥25K ops with 2048 slots → Crashes immediately
• Pattern: Scales with working set size (more live objects = more failures)

---

Technical Analysis

1. Allocation Flow (Working)

malloc(size) [size ≤ 1KB]
 ↓
hak_alloc_at(size)
 ↓
hak_tiny_alloc_fast_wrapper(size)
 ↓
tiny_alloc_fast(size)
 ↓ [TLS freelist miss]
 ↓
hak_tiny_alloc_slow(size)
 ↓
hak_tiny_alloc_superslab(class_idx)
 ↓
✅ Returns pointer WITHOUT header (SuperSlab allocation)

2. Free Flow (Broken)

free(ptr)
 ↓
hak_free_at(ptr, 0, site)
 ↓
[SS-first free path] hak_super_lookup(ptr)
 ↓ ❌ Lookup FAILS (should succeed!)
 ↓
[Fallback] Try mid/L25 lookup → Fails
 ↓
[Fallback] Header dispatch:
    void* raw = (char*)ptr - HEADER_SIZE;  // ← ptr has NO header!
    AllocHeader* hdr = (AllocHeader*)raw;   // ← Invalid pointer
    if (hdr->magic != HAKMEM_MAGIC) {      // ← ⚠️ SEGV or reads 0x0
        // g_invalid_free_mode = 1 (direct-link)
        goto done;  // ← ❌ MEMORY LEAK!
    }

Key Bug: When SuperSlab lookup fails for a tiny allocation, the code assumes there's a HAKMEM header and tries to read it. But tiny allocations are headerless, so this reads invalid memory.

3. Why SuperSlab Lookup Fails

Based on testing:

# Default (crashes with "Invalid magic 0x0")
./bench_random_mixed_hakmem 25000 2048 123
# → Hundreds of "Invalid magic" errors

# With SuperSlab explicitly enabled (no "Invalid magic" errors, but still SEGVs)
HAKMEM_TINY_USE_SUPERSLAB=1 ./bench_random_mixed_hakmem 25000 2048 123
# → SEGV without "Invalid magic" errors

Hypothesis: When HAKMEM_TINY_USE_SUPERSLAB is not explicitly set, there may be a code path where:

1. Tiny allocations succeed (from some non-SuperSlab path)
2. But they're not registered in the SuperSlab registry
3. So lookups fail during free

Possible causes:

• Configuration bug: g_use_superslab may be uninitialized or overridden
• TLS allocation path: There may be a TLS-only allocation path that bypasses SuperSlab
• Magazine/HotMag path: Allocations from magazine layers might not come from SuperSlab
• Registry capacity: Registry might be full (unlikely with SUPER_REG_SIZE=262144)

4. Direct-Link vs LD_PRELOAD Behavior

LD_PRELOAD (hak_core_init.inc.h:147-164):

if (ldpre && strstr(ldpre, "libhakmem.so")) {
    g_ldpreload_mode = 1;
    g_invalid_free_mode = 0;  // ← Fallback to libc
}

• Defaults to g_invalid_free_mode=0 (fallback mode)
• Invalid frees → __libc_free(ptr) → masks the bug (may work if ptr was originally from libc)

Direct-Link:

else {
    g_invalid_free_mode = 1;  // ← Skip invalid frees
}

• Defaults to g_invalid_free_mode=1 (skip mode)
• Invalid frees → goto done → silent memory leak
• Accumulated leaks → memory exhaustion → SEGV

---

GDB Analysis

Backtrace

Thread 1 "bench_random_mi" received signal SIGSEGV, Segmentation fault.
0x000055555555eb40 in free ()

#0  0x000055555555eb40 in free ()
#1  0xffffffffffffffff in ?? ()
...
#8  0x00005555555587e1 in main ()

Registers:
rax  0x555556c9d040   (some address)
rbp  0x7ffff6e00000   (pointer being freed - page-aligned!)
rdi  0x0               (NULL!)
rip  0x55555555eb40   <free+2176>

Disassembly at Crash Point (free+2176)

0xab40 <+2176>:  mov  -0x28(%rbp),%ecx      # Load header magic
0xab43 <+2179>:  cmp  $0x48414B4D,%ecx      # Compare with HAKMEM_MAGIC
0xab49 <+2185>:  je   0xabd0 <free+2320>    # Jump if magic matches

Key observation:

• rbp = 0x7ffff6e00000 (page-aligned, likely start of mmap region)
• Trying to read from rbp - 0x28 = 0x7ffff6dffffd8
• If this is at page boundary, reading before the page causes SEGV

---

Proposed Fix

Option A: Safe Header Read (Recommended)

Add a safety check before reading the header:

// hak_free_api.inc.h, line 78-88 (header dispatch)

// BEFORE: Unsafe header read
void* raw = (char*)ptr - HEADER_SIZE;
AllocHeader* hdr = (AllocHeader*)raw;
if (hdr->magic != HAKMEM_MAGIC) { ... }

// AFTER: Safe fallback for tiny allocations
// If SuperSlab lookup failed for a tiny-sized allocation,
// assume it's an invalid free or was already freed
{
    // Check if this could be a tiny allocation (size ≤ 1KB)
    // Heuristic: If SuperSlab/Mid/L25 lookup all failed, and we're here,
    // either it's a libc allocation with header, or a leaked tiny allocation

    // Try to safely read header magic
    void* raw = (char*)ptr - HEADER_SIZE;
    AllocHeader* hdr = (AllocHeader*)raw;

    // If magic is valid, proceed with header dispatch
    if (hdr->magic == HAKMEM_MAGIC) {
        // Header exists, dispatch normally
        if (HAK_ENABLED_CACHE(HAKMEM_FEATURE_BIGCACHE) && hdr->class_bytes >= 2097152) {
            if (hak_bigcache_put(ptr, hdr->size, hdr->alloc_site)) goto done;
        }
        switch (hdr->method) {
            case ALLOC_METHOD_MALLOC: __libc_free(raw); break;
            case ALLOC_METHOD_MMAP: /* ... */ break;
            // ...
        }
    } else {
        // Invalid magic - could be:
        // 1. Tiny allocation where SuperSlab lookup failed
        // 2. Already freed pointer
        // 3. Pointer from external library

        if (g_invalid_free_log) {
            fprintf(stderr, "[hakmem] WARNING: free() of pointer %p with invalid magic 0x%X (expected 0x%X)\n",
                    ptr, hdr->magic, HAKMEM_MAGIC);
            fprintf(stderr, "[hakmem] Possible causes: tiny allocation lookup failure, double-free, or external pointer\n");
        }

        // In direct-link mode, do NOT leak - try to return to tiny pool
        // as a best-effort recovery
        if (!g_ldpreload_mode) {
            // Attempt to route to tiny free (may succeed if it's a valid tiny allocation)
            hak_tiny_free(ptr);  // Will validate internally
        } else {
            // LD_PRELOAD mode: fallback to libc (may be mixed allocation)
            if (g_invalid_free_mode == 0) {
                __libc_free(ptr);  // Not raw! ptr itself
            }
        }
    }
}
goto done;

Option B: Fix SuperSlab Lookup Root Cause

Investigate why SuperSlab lookups are failing:

1. Add comprehensive logging:

// At allocation time
fprintf(stderr, "[ALLOC_DEBUG] ptr=%p class=%d from_superslab=%d\n",
        ptr, class_idx, from_superslab);

// At free time
SuperSlab* ss = hak_super_lookup(ptr);
fprintf(stderr, "[FREE_DEBUG] ptr=%p lookup=%p magic=%llx\n",
        ptr, ss, ss ? ss->magic : 0);

2. Check TLS allocation paths:

• Verify all paths through tiny_alloc_fast_pop() come from SuperSlab
• Check if magazine/HotMag allocations are properly registered
• Verify TLS SLL allocations are from registered SuperSlabs

3. Verify registry initialization:

// At startup
fprintf(stderr, "[INIT] g_super_reg_initialized=%d g_use_superslab=%d\n",
        g_super_reg_initialized, g_use_superslab);

Option C: Force SuperSlab Path

Simplify the allocation path to always use SuperSlab:

// Disable competing paths that might bypass SuperSlab
g_hotmag_enable = 0;       // Disable HotMag
g_tls_list_enable = 0;     // Disable TLS List
g_tls_sll_enable = 1;      // Enable TLS SLL (SuperSlab-backed)

---

Immediate Workaround

For users hitting this bug:

# Workaround 1: Use LD_PRELOAD (masks the issue)
LD_PRELOAD=./libhakmem.so your_benchmark

# Workaround 2: Force SuperSlab (may still crash, but different symptoms)
HAKMEM_TINY_USE_SUPERSLAB=1 ./your_benchmark

# Workaround 3: Disable tiny allocator (fallback to libc)
HAKMEM_WRAP_TINY=0 ./your_benchmark

---

Next Steps

1. Implement Option A (Safe Header Read) - Immediate fix to prevent SEGV
2. Add logging to identify root cause - Why are SuperSlab lookups failing?
3. Fix underlying issue - Ensure all tiny allocations are SuperSlab-backed
4. Add regression tests - Prevent future breakage

---

Files to Modify

1. /mnt/workdisk/public_share/hakmem/core/box/hak_free_api.inc.h - Lines 78-120 (header dispatch logic)
2. /mnt/workdisk/public_share/hakmem/core/hakmem_tiny.c - Add allocation path logging
3. /mnt/workdisk/public_share/hakmem/core/tiny_alloc_fast.inc.h - Verify SuperSlab usage
4. /mnt/workdisk/public_share/hakmem/core/hakmem_super_registry.c - Add lookup diagnostics

---

Related Issues

• Phase 6-2.3: Active counter bug fix (freed blocks not tracked)
• Sanitizer Fix: Similar TLS initialization ordering issues
• LD_PRELOAD vs Direct-Link: Behavioral differences in error handling

---

Verification

After fix, verify:

# Should complete without errors
./bench_random_mixed_hakmem 50000 2048 123
./bench_mid_large_mt_hakmem 4 40000 2048 42

# Should see no "Invalid magic" errors
HAKMEM_INVALID_FREE_LOG=1 ./bench_random_mixed_hakmem 50000 2048 123