hakmem/docs/analysis/FALSE_POSITIVE_SEGV_FIX.md

# FINAL FIX: Header Magic SEGV (2025-11-07)

## Problem Analysis

### Root Cause
SEGV at `core/box/hak_free_api.inc.h:115` when dereferencing `hdr->magic`:

```c
void* raw = (char*)ptr - HEADER_SIZE;  // Line 113
AllocHeader* hdr = (AllocHeader*)raw;   // Line 114
if (hdr->magic != HAKMEM_MAGIC) {      // Line 115 ← SEGV HERE
```

**Why it crashes:**
- `ptr` might be from Tiny SuperSlab (no header) where SS lookup failed
- `ptr` might be from libc (in mixed environments)
- `raw = ptr - HEADER_SIZE` points to unmapped/invalid memory
- Dereferencing `hdr->magic` → **SEGV**

### Evidence
```bash
# Works (all Tiny 8-128B, caught by SS-first)
./larson_hakmem 10 8 128 1024 1 12345 4
→ 838K ops/s ✅

# Crashes (mixed sizes, some escape SS lookup)
./bench_random_mixed_hakmem 50000 2048 1234567
→ SEGV (Exit 139) ❌
```

## Solution: Safe Memory Access Check

### Approach
Use a **lightweight memory accessibility check** before dereferencing the header.

**Why not other approaches?**
- ❌ Signal handlers: Complex, non-portable, huge overhead
- ❌ Page alignment: Doesn't guarantee validity
- ❌ Reorder logic only: Doesn't solve unmapped memory dereference
- ✅ **Memory check + fallback**: Safe, minimal, predictable

### Implementation

#### Option 1: mincore() (Recommended)
**Pros:** Portable, reliable, acceptable overhead (only on fallback path)
**Cons:** System call (but only when all lookups fail)

```c
// Add to core/hakmem_internal.h
static inline int hak_is_memory_readable(void* addr) {
    #ifdef __linux__
    unsigned char vec;
    // mincore returns 0 if page is mapped, -1 (ENOMEM) if not
    return mincore(addr, 1, &vec) == 0;
    #else
    // Fallback: assume accessible (conservative)
    return 1;
    #endif
}
```

#### Option 2: msync() (Alternative)
**Pros:** Also portable, checks if memory is valid
**Cons:** Slightly more overhead

```c
static inline int hak_is_memory_readable(void* addr) {
    #ifdef __linux__
    // msync with MS_ASYNC is lightweight check
    return msync(addr, 1, MS_ASYNC) == 0 || errno == ENOMEM;
    #else
    return 1;
    #endif
}
```

#### Modified Free Path

```c
// core/box/hak_free_api.inc.h lines 111-151
// Replace lines 113-151 with:

{
    void* raw = (char*)ptr - HEADER_SIZE;

    // CRITICAL FIX: Check if memory is accessible before dereferencing
    if (!hak_is_memory_readable(raw)) {
        // Memory not accessible, ptr likely has no header (Tiny or libc)
        hak_free_route_log("unmapped_header_fallback", ptr);

        // In direct-link mode, try tiny_free (handles headerless Tiny allocs)
        if (!g_ldpreload_mode && g_invalid_free_mode) {
            hak_tiny_free(ptr);
            goto done;
        }

        // LD_PRELOAD mode: route to libc (might be libc allocation)
        extern void __libc_free(void*);
        __libc_free(ptr);
        goto done;
    }

    // Safe to dereference header now
    AllocHeader* hdr = (AllocHeader*)raw;

    // Check magic number
    if (hdr->magic != HAKMEM_MAGIC) {
        // Invalid magic (existing error handling)
        if (g_invalid_free_log) fprintf(stderr, "[hakmem] ERROR: Invalid magic 0x%X (expected 0x%X)\n", hdr->magic, HAKMEM_MAGIC);
        hak_super_reg_reqtrace_dump(ptr);

        if (!g_ldpreload_mode && g_invalid_free_mode) {
            hak_free_route_log("invalid_magic_tiny_recovery", ptr);
            hak_tiny_free(ptr);
            goto done;
        }

        if (g_invalid_free_mode) {
            static int leak_warn = 0;
            if (!leak_warn) {
                fprintf(stderr, "[hakmem] WARNING: Skipping free of invalid pointer %p (may leak memory)\n", ptr);
                leak_warn = 1;
            }
            goto done;
        } else {
            extern void __libc_free(void*);
            __libc_free(ptr);
            goto done;
        }
    }

    // Valid header, proceed with normal dispatch
    if (HAK_ENABLED_CACHE(HAKMEM_FEATURE_BIGCACHE) && hdr->class_bytes >= 2097152) {
        if (hak_bigcache_put(ptr, hdr->size, hdr->alloc_site)) goto done;
    }
    {
        static int g_bc_l25_en_free = -1; if (g_bc_l25_en_free == -1) { const char* e = getenv("HAKMEM_BIGCACHE_L25"); g_bc_l25_en_free = (e && atoi(e) != 0) ? 1 : 0; }
        if (g_bc_l25_en_free && HAK_ENABLED_CACHE(HAKMEM_FEATURE_BIGCACHE) && hdr->size >= 524288 && hdr->size < 2097152) {
            if (hak_bigcache_put(ptr, hdr->size, hdr->alloc_site)) goto done;
        }
    }
    switch (hdr->method) {
        case ALLOC_METHOD_POOL: if (HAK_ENABLED_ALLOC(HAKMEM_FEATURE_POOL)) { hkm_ace_stat_mid_free(); hak_pool_free(ptr, hdr->size, hdr->alloc_site); goto done; } break;
        case ALLOC_METHOD_L25_POOL: hkm_ace_stat_large_free(); hak_l25_pool_free(ptr, hdr->size, hdr->alloc_site); goto done;
        case ALLOC_METHOD_MALLOC:
            hak_free_route_log("malloc_hdr", ptr);
            extern void __libc_free(void*);
            __libc_free(raw);
            break;
        case ALLOC_METHOD_MMAP:
#ifdef __linux__
            if (HAK_ENABLED_MEMORY(HAKMEM_FEATURE_BATCH_MADVISE) && hdr->size >= BATCH_MIN_SIZE) { hak_batch_add(raw, hdr->size); goto done; }
            if (hkm_whale_put(raw, hdr->size) != 0) { hkm_sys_munmap(raw, hdr->size); }
#else
            extern void __libc_free(void*);
            __libc_free(raw);
#endif
            break;
        default: fprintf(stderr, "[hakmem] ERROR: Unknown allocation method: %d\n", hdr->method); break;
    }
}
```

## Performance Impact

### Overhead Analysis
- **mincore()**: ~50-100 cycles (system call)
- **Only triggered**: When all lookups fail (SS, Mid, L25)
- **Typical case**: Never reached (lookups succeed)
- **Failure case**: Acceptable overhead vs SEGV

### Benchmark Predictions
```
Larson (all Tiny):       No impact (SS-first catches all)
Random Mixed (varied):   +0-2% overhead (rare fallback)
Worst case (all miss):   +5-10% (but prevents SEGV)
```

## Verification Steps

### Step 1: Apply Fix
```bash
# Edit core/hakmem_internal.h (add helper function)
# Edit core/box/hak_free_api.inc.h (add memory check)
```

### Step 2: Rebuild
```bash
make clean
make bench_random_mixed_hakmem larson_hakmem
```

### Step 3: Test
```bash
# Test 1: Larson (should still work)
./larson_hakmem 10 8 128 1024 1 12345 4
# Expected: ~838K ops/s ✅

# Test 2: Random Mixed (should no longer crash)
./bench_random_mixed_hakmem 50000 2048 1234567
# Expected: Completes without SEGV ✅

# Test 3: Stress test
for i in {1..100}; do
    ./bench_random_mixed_hakmem 10000 2048 $i || echo "FAIL: $i"
done
# Expected: All pass ✅
```

### Step 4: Performance Check
```bash
# Verify no regression on Larson
./larson_hakmem 2 8 128 1024 1 12345 4
# Should be similar to baseline (4.19M ops/s)

# Check random_mixed performance
./bench_random_mixed_hakmem 100000 2048 1234567
# Should complete successfully with reasonable performance
```

## Alternative: Root Cause Fix (Future Work)

The memory check fix is **safe and minimal**, but the root cause is:
**Registry lookups are not catching all allocations.**

Future investigation:
1. Why do Tiny allocations escape SS registry?
2. Are Mid/L25 registries populated correctly?
3. Thread safety of registry operations?

### Investigation Commands
```bash
# Enable registry trace
HAKMEM_SUPER_REG_REQTRACE=1 ./bench_random_mixed_hakmem 1000 2048 1234567

# Enable free route trace
HAKMEM_FREE_ROUTE_TRACE=1 ./bench_random_mixed_hakmem 1000 2048 1234567
```

## Summary

### The Fix
✅ **Add memory accessibility check before header dereference**
- Minimal code change (10 lines)
- Safe and portable
- Acceptable performance impact
- Prevents all unmapped memory dereferences

### Why This Works
1. **Detects unmapped memory** before dereferencing
2. **Routes to correct handler** (tiny_free or libc_free)
3. **No false positives** (mincore is reliable)
4. **Preserves existing logic** (only adds safety check)

### Expected Outcome
```
Before: SEGV on bench_random_mixed
After:  Completes successfully
Performance: ~0-2% overhead (acceptable)
```