hakmem/docs/analysis/POOL_TLS_SEGV_ROOT_CAUSE.md

Pool TLS Phase 1.5a SEGV - TRUE ROOT CAUSE

Executive Summary

ACTUAL ROOT CAUSE: Missing Object Files in Link Command

The SEGV was NOT caused by TLS initialization ordering or uninitialized variables. It was caused by undefined references to pool_alloc() and pool_free() because the Pool TLS object files were not included in the link command.

What Actually Happened

Build Evidence:

# Without POOL_TLS_PHASE1=1 make variable:
$ make bench_random_mixed_hakmem
/usr/bin/ld: undefined reference to `pool_alloc'
/usr/bin/ld: undefined reference to `pool_free'
collect2: error: ld returned 1 exit status

# With POOL_TLS_PHASE1=1 make variable:
$ make bench_random_mixed_hakmem POOL_TLS_PHASE1=1
# Links successfully! ✅

Makefile Analysis

File: /mnt/workdisk/public_share/hakmem/Makefile:319-323

TINY_BENCH_OBJS = $(TINY_BENCH_OBJS_BASE)
ifeq ($(POOL_TLS_PHASE1),1)
TINY_BENCH_OBJS += pool_tls.o pool_refill.o core/pool_tls_arena.o
endif

Problem:

• Lines 150-151 enable HAKMEM_POOL_TLS_PHASE1=1 in CFLAGS (unconditionally)
• But Makefile line 321 checks $(POOL_TLS_PHASE1) variable (NOT defined!)
• Result: Code compiles with #ifdef HAKMEM_POOL_TLS_PHASE1 enabled, but object files NOT linked

Why This Caused Confusion

Three layers of confusion:

1. CFLAGS vs Make Variable Mismatch:

   • CFLAGS += -DHAKMEM_POOL_TLS_PHASE1=1 (line 150) → Code compiles with Pool TLS enabled
   • ifeq ($(POOL_TLS_PHASE1),1) (line 321) → Checks undefined Make variable → False
   • Result: Conditional compilation YES, conditional linking NO

2. Linker Error Looked Like Runtime SEGV:

   • User reported "SEGV (Exit 139)"
   • This was likely the linker error exit code, not a runtime SEGV!
   • No binary was produced, so there was no runtime crash

3. Debug Prints Never Appeared:

   • User added fprintf() to hak_free_api.inc.h:145-146
   • Binary never built (linker error) → old binary still existed
   • Running old binary → debug prints don't appear → looks like crash happens before that line

Verification

Built with correct Make variable:

$ make bench_random_mixed_hakmem POOL_TLS_PHASE1=1
gcc -o bench_random_mixed_hakmem ... pool_tls.o pool_refill.o core/pool_tls_arena.o ...
# ✅ SUCCESS!

$ ./bench_random_mixed_hakmem 1000 8192 1234567
[Pool] hak_pool_init() called for the first time
# ✅ RUNS WITHOUT SEGV!

What The GDB Evidence Actually Meant

User's GDB output:

(gdb) p $rbp
$1 = (void *) 0x7ffff7137017

(gdb) p $rdi
$2 = 0

Crash instruction: movzbl -0x1(%rbp),%edx

Re-interpretation:

• This was from running an OLD binary (before Pool TLS was added)
• The old binary crashed on some unrelated code path
• User thought it was Pool TLS-related because they were trying to test Pool TLS
• Actual crash: Unrelated to Pool TLS (old code bug)

The Fix

Option A: Set POOL_TLS_PHASE1 Make variable (QUICK FIX - DONE):

make bench_random_mixed_hakmem POOL_TLS_PHASE1=1

Option B: Remove conditional (if always enabled):

 # Makefile:319-323
 TINY_BENCH_OBJS = $(TINY_BENCH_OBJS_BASE)
-ifeq ($(POOL_TLS_PHASE1),1)
 TINY_BENCH_OBJS += pool_tls.o pool_refill.o core/pool_tls_arena.o
-endif

Option C: Auto-detect from CFLAGS:

# Auto-detect if HAKMEM_POOL_TLS_PHASE1 is in CFLAGS
ifneq (,$(findstring -DHAKMEM_POOL_TLS_PHASE1=1,$(CFLAGS)))
TINY_BENCH_OBJS += pool_tls.o pool_refill.o core/pool_tls_arena.o
endif

Why My Initial Investigation Was Wrong

I made these assumptions:

1. Binary was built successfully (it wasn't - linker error!)
2. SEGV was runtime crash (it was linker error or old binary crash!)
3. TLS variables were being accessed (they weren't - code never linked!)
4. Debug prints should appear (they couldn't - new code never built!)

Lesson learned:

• Always check linker output, not just compiler warnings
• Verify binary timestamp matches source changes
• Don't trust runtime behavior when build might have failed

Current Status

Pool TLS Phase 1.5a: WORKS! ✅

$ make clean && make bench_random_mixed_hakmem POOL_TLS_PHASE1=1
$ ./bench_random_mixed_hakmem 1000 8192 1234567
# Runs successfully, no SEGV!

Recommended Actions

1. Immediate (DONE):

   • Document: Users must build with POOL_TLS_PHASE1=1 make variable

2. Short-term (1 hour):

   • Update Makefile to remove conditional or auto-detect from CFLAGS

3. Long-term (Optional):

   • Add build verification script (check that binary contains expected symbols)
   • Add Makefile warning if CFLAGS and Make variables mismatch

Apology

My initial 3000-line investigation report was completely wrong. The issue was a simple Makefile variable mismatch, not a complex TLS initialization ordering problem.

Key takeaways:

• Always verify the build succeeded before investigating runtime behavior
• Check linker errors first (undefined references = missing object files)
• Don't overthink when the answer is simple

---

Investigation completed: 2025-11-09 True root cause: Makefile conditional mismatch (CFLAGS vs Make variable) Fix: Build with POOL_TLS_PHASE1=1 or remove conditional Status: Pool TLS Phase 1.5a WORKING ✅