hakmem/docs/analysis/FREE_PATH_INVESTIGATION.md

Free Path Freelist Push Investigation

Executive Summary

Investigation of the same-thread free path for freelist push implementation has identified ONE CRITICAL BUG and MULTIPLE DESIGN ISSUES that explain the freelist reuse rate problem.

Critical Finding: The freelist push is being performed, but it is only visible when blocks are accessed from the refill path, not when they're accessed from normal allocation paths. This creates a visibility gap in the publish/fetch mechanism.

---

Investigation Flow: free() → alloc()

Phase 1: Same-Thread Free (freelist push)

File: core/hakmem_tiny_free.inc (lines 1-608) Main Function: hak_tiny_free_superslab(void* ptr, SuperSlab* ss) (lines ~150-300)

Fast Path Decision (Line 121):

if (!g_tiny_force_remote && meta->owner_tid != 0 && meta->owner_tid == my_tid) {
    // Same-thread free
    // ...
    tiny_free_local_box(ss, slab_idx, meta, ptr, my_tid);

Status: ✓ CORRECT - ownership check is present

Freelist Push Implementation

File: core/box/free_local_box.c (lines 5-36)

void tiny_free_local_box(SuperSlab* ss, int slab_idx, TinySlabMeta* meta, void* ptr, uint32_t my_tid) {
    void* prev = meta->freelist;
    *(void**)ptr = prev;
    meta->freelist = ptr;              // <-- FREELIST PUSH HAPPENS HERE (Line 12)
    
    // ...
    meta->used--;
    ss_active_dec_one(ss);
    
    if (prev == NULL) {
        // First-free → publish
        tiny_free_publish_first_free((int)ss->size_class, ss, slab_idx);  // Line 34
    }
}

Status: ✓ CORRECT - freelist push happens unconditionally before publish

Publish Mechanism

File: core/box/free_publish_box.c (lines 23-28)

void tiny_free_publish_first_free(int class_idx, SuperSlab* ss, int slab_idx) {
    tiny_ready_push(class_idx, ss, slab_idx);
    ss_partial_publish(class_idx, ss);
    mailbox_box_publish(class_idx, ss, slab_idx);  // Line 28
}

File: core/box/mailbox_box.c (lines 112-122)

void mailbox_box_publish(int class_idx, SuperSlab* ss, int slab_idx) {
    mailbox_box_register(class_idx);
    uintptr_t ent = ((uintptr_t)ss) | ((uintptr_t)slab_idx & 0x3Fu);
    uint32_t slot = g_tls_mailbox_slot[class_idx];
    atomic_store_explicit(&g_pub_mailbox_entries[class_idx][slot], ent, memory_order_release);
    g_pub_mail_hits[class_idx]++;  // Line 122 - COUNTER INCREMENTED
}

Status: ✓ CORRECT - publish happens on first-free

---

Phase 2: Refill/Adoption Path (mailbox fetch)

File: core/tiny_refill.h (lines 136-157)

// For hot tiny classes (0..3), try mailbox first
if (class_idx <= 3) {
    uint32_t self_tid = tiny_self_u32();
    ROUTE_MARK(3);
    uintptr_t mail = mailbox_box_fetch(class_idx);  // Line 139
    if (mail) {
        SuperSlab* mss = slab_entry_ss(mail);
        int midx = slab_entry_idx(mail);
        SlabHandle h = slab_try_acquire(mss, midx, self_tid);
        if (slab_is_valid(&h)) {
            if (slab_remote_pending(&h)) {
                slab_drain_remote_full(&h);
            } else if (slab_freelist(&h)) {
                tiny_tls_bind_slab(tls, h.ss, h.slab_idx);
                ROUTE_MARK(4);
                return h.ss;  // Success!
            }
        }
    }
}

Status: ✓ CORRECT - mailbox fetch is called for refill

Mailbox Fetch Implementation

File: core/box/mailbox_box.c (lines 160-207)

uintptr_t mailbox_box_fetch(int class_idx) {
    uint32_t used = atomic_load_explicit(&g_pub_mailbox_used[class_idx], memory_order_acquire);
    
    // Destructive fetch of first available entry (0..used-1)
    for (uint32_t i = 0; i < used; i++) {
        uintptr_t ent = atomic_exchange_explicit(&g_pub_mailbox_entries[class_idx][i],
                                                 (uintptr_t)0,
                                                 memory_order_acq_rel);
        if (ent) {
            g_rf_hit_mail[class_idx]++;  // Line 200 - COUNTER INCREMENTED
            return ent;
        }
    }
    return (uintptr_t)0;
}

---

## Fix Log (2025-11-06)

- P0: nonempty_maskをクリアしない
  - 変更: `core/slab_handle.h` の `slab_freelist_pop()` で `nonempty_mask` を空→空転でクリアする処理を削除。
  - 理由: 一度でも非空になった slab を再発見できるようにして、free後の再利用が見えなくなるリークを防止。

- P0: adopt_gate の TOCTOU 安全化
  - 変更: すべての bind 直前の判定を `slab_is_safe_to_bind()` に統一。`core/tiny_refill.h` の mailbox/hot/ready/BG 集約の分岐を更新。
  - 変更: adopt_gate 実装側（`core/hakmem_tiny.c`）は `slab_drain_remote_full()` の後に `slab_is_safe_to_bind()` を必ず最終確認。

- P1: Refill アイテム内訳カウンタの追加
  - 変更: `core/hakmem_tiny.c` に `g_rf_freelist_items[]` / `g_rf_carve_items[]` を追加。
  - 変更: `core/hakmem_tiny_refill_p0.inc.h` で freelist/carve 取得数をカウント。
  - 変更: `core/hakmem_tiny_stats.c` のダンプに [Refill Item Sources] を追加。

- Mailbox 実装の一本化
  - 変更: 旧 `core/tiny_mailbox.c/.h` を削除。実装は `core/box/mailbox_box.*` のみ（包括的な Box）に統一。

- Makefile 修正
  - 変更: タイポ修正 `>/devnull` → `>/dev/null`。

### 検証の目安（SIGUSR1/終了時ダンプ）

- [Refill Stage] の mail/reg/ready が 0 のままになっていないか
- [Refill Item Sources] で freelist/carve のバランス（freelist が上がれば再利用が通電）
- [Publish Hits] / [Publish Pipeline] が 0 連発のときは、`HAKMEM_TINY_FREE_TO_SS=1` や `HAKMEM_TINY_FREELIST_MASK=1` を一時有効化

Status: ✓ CORRECT - fetch clears the mailbox entry

---

Critical Bug Found

BUG #1: Freelist Access Without Publish

Location: core/hakmem_tiny_free.inc (lines 687-695) Function: superslab_alloc_from_slab() - Direct freelist pop during allocation

// Freelist mode (after first free())
if (meta->freelist) {
    void* block = meta->freelist;
    meta->freelist = *(void**)block;  // Pop from freelist
    meta->used++;
    tiny_remote_track_on_alloc(ss, slab_idx, block, "freelist_alloc", 0);
    tiny_remote_assert_not_remote(ss, slab_idx, block, "freelist_alloc_ret", 0);
    return block;  // Direct pop - NO mailbox tracking!
}

Problem: When allocation directly pops from meta->freelist, it completely bypasses the mailbox layer. This means:

1. Block is pushed to freelist via tiny_free_local_box() ✓
2. Mailbox is published on first-free ✓
3. But if the block is accessed during direct freelist pop, the mailbox entry is never fetched or cleared
4. The mailbox entry remains stale, wasting a slot permanently

Impact:

• Permanent mailbox slot leakage - Published blocks that are directly popped are never cleared
• False positive in g_pub_mail_hits[] - count includes blocks that bypassed the fetch path
• Freelist reuse becomes invisible to refill metrics because it doesn't go through mailbox_box_fetch()

BUG #2: Premature Publish Before Freelist Formation

Location: core/box/free_local_box.c (lines 32-34) Issue: Publish happens only on first-free (prev==NULL)

if (prev == NULL) {
    tiny_free_publish_first_free((int)ss->size_class, ss, slab_idx);
}

Problem: Once first-free publishes, subsequent pushes (prev!=NULL) are silent:

• Block 1 freed: freelist=[1], mailbox published ✓
• Block 2 freed: freelist=[2→1], mailbox NOT updated ⚠️
• Block 3 freed: freelist=[3→2→1], mailbox NOT updated ⚠️

The mailbox only ever contains the first freed block in the slab. If that block is allocated and then freed again, the mailbox entry is not refreshed.

Impact:

• Freelist state changes after first-free are not advertised
• Refill can't discover newly available blocks without full registry scan
• Forces slower adoption path (registry scan) instead of mailbox hit

---

Design Issues

Issue #1: Missing Freelist State Visibility

The core problem: Meta->freelist is not synchronized with publish state.

Current Flow:

free()
  → tiny_free_local_box()
      → meta->freelist = ptr  (direct write, no sync)
      → if (prev==NULL) mailbox_publish()  (one-time)

refill()
  → Try mailbox_box_fetch()  (gets only first-free block)
  → If miss, scan registry   (slow path, O(n))
  → If found, adopt & pop freelist
  
alloc()
  → superslab_alloc_from_slab()
      → if (meta->freelist) pop  (direct access, bypasses mailbox!)

Missing: Mailbox consistency check when freelist is accessed

Issue #2: Adoption vs. Direct Access Race

Location: core/hakmem_tiny_free.inc (line 687-695)

Thread A: Thread B:

1. Allocate from SS
2. Free block → freelist=[1]
3. Publish mailbox ✓
   4. Refill: Try adopt 5. Mailbox fetch gets [1] ✓ 6. Ownership acquire → success 7. But direct alloc bypasses this path!
4. Alloc again (same thread)
5. Pop from freelist directly → mailbox entry stale now

Result: Mailbox state diverges from actual freelist state

Issue #3: Ownership Transition Not Tracked

When meta->owner_tid changes (cross-thread ownership transfer), freelist is not re-published:

Location: core/hakmem_tiny_free.inc (lines 120-135)

if (!g_tiny_force_remote && meta->owner_tid != 0 && meta->owner_tid == my_tid) {
    // Same-thread path
} else {
    // Cross-thread path - but NO REPUBLISH if ownership changes
}

Missing: When ownership transitions to a new thread, the existing freelist should be advertised to that thread

---

Metrics Analysis

The counters reveal the issue:

In core/box/mailbox_box.c (Line 122):

void mailbox_box_publish(int class_idx, SuperSlab* ss, int slab_idx) {
    // ...
    g_pub_mail_hits[class_idx]++;  // Published count
}

In core/box/mailbox_box.c (Line 200):

uintptr_t mailbox_box_fetch(int class_idx) {
    if (ent) {
        g_rf_hit_mail[class_idx]++;  // Fetched count
        return ent;
    }
    return (uintptr_t)0;
}

Expected Relationship: g_rf_hit_mail[class_idx] should be ~1.0x of g_pub_mail_hits[class_idx] Actual Relationship: Probably 0.1x - 0.5x (many published entries never fetched)

Explanation:

• Blocks are published (g_pub_mail_hits++)
• But they're accessed via direct freelist pop (no fetch)
• So g_rf_hit_mail stays low
• Mailbox entries accumulate as garbage

---

Root Cause Summary

Root Cause: The freelist push is functional, but the visibility mechanism (mailbox) is decoupled from the actual freelist access pattern.

The system assumes refill always goes through mailbox_fetch(), but direct freelist pops bypass this entirely, creating:

1. Stale mailbox entries - Published but never fetched
2. Invisible reuse - Freed blocks are reused directly without fetch visibility
3. Metric misalignment - g_pub_mail_hits >> g_rf_hit_mail

---

Recommended Fixes

Fix #1: Clear Stale Mailbox Entry on Direct Pop

File: core/hakmem_tiny_free.inc (lines 687-695) In: superslab_alloc_from_slab()

if (meta->freelist) {
    void* block = meta->freelist;
    meta->freelist = *(void**)block;
    meta->used++;
    
    // NEW: If this is a mailbox-published slab, clear the entry
    if (slab_idx == 0) {  // Only first slab publishes
        // Signal to refill: this slab's mailbox entry may now be stale
        // Option A: Mark as dirty (requires new field)
        // Option B: Clear mailbox on first pop (requires sync)
    }
    
    return block;
}

Fix #2: Republish After Each Free (Aggressive)

File: core/box/free_local_box.c (lines 32-34) Problem: Only first-free publishes

Change:

// Always publish if freelist is non-empty
if (meta->freelist != NULL) {
    tiny_free_publish_first_free((int)ss->size_class, ss, slab_idx);
}

Cost: More atomic operations, but ensures mailbox is always up-to-date

Fix #3: Track Freelist Modifications via Atomic

New Approach: Use atomic freelist_mask as published state

File: core/box/free_local_box.c (current lines 15-25)

// Already implemented - use this more aggressively
if (prev == NULL) {
    uint32_t bit = (1u << slab_idx);
    atomic_fetch_or_explicit(&ss->freelist_mask, bit, memory_order_release);
}

// Also mark on later frees
else {
    uint32_t bit = (1u << slab_idx);
    atomic_fetch_or_explicit(&ss->freelist_mask, bit, memory_order_release);
}

Fix #4: Add Freelist Consistency Check in Refill

File: core/tiny_refill.h (lines ~140-156) New Logic:

uintptr_t mail = mailbox_box_fetch(class_idx);
if (mail) {
    SuperSlab* mss = slab_entry_ss(mail);
    int midx = slab_entry_idx(mail);
    SlabHandle h = slab_try_acquire(mss, midx, self_tid);
    if (slab_is_valid(&h)) {
        if (slab_freelist(&h)) {
            // NEW: Verify mailbox entry matches actual freelist
            if (h.ss->slabs[h.slab_idx].freelist == NULL) {
                // Stale entry - was already popped directly
                // Re-publish if more blocks freed since
                continue;  // Try next candidate
            }
            tiny_tls_bind_slab(tls, h.ss, h.slab_idx);
            return h.ss;
        }
    }
}

---

Testing Recommendations

Test 1: Mailbox vs. Direct Pop Ratio

Instrument the code to measure:

• mailbox_fetch_calls vs direct_freelist_pops
• Expected ratio after warmup: Should be ~1:1 if refill path is being used
• Actual ratio: Probably 1:10 or worse (direct pops dominating)

Test 2: Mailbox Entry Staleness

Enable debug mode and check:

HAKMEM_TINY_MAILBOX_TRACE=1 HAKMEM_TINY_RF_TRACE=1 ./larson

Examine MBTRACE output:

• Count "publish" events vs "fetch" events
• Any publish without matching fetch = wasted slot

Test 3: Freelist Reuse Path

Add instrumentation to superslab_alloc_from_slab():

if (meta->freelist) {
    g_direct_freelist_pops[class_idx]++;  // New counter
}

Compare with refill path:

g_refill_calls[class_idx]++;

Verify that most allocations come from direct freelist (expected) vs. refill (if low, freelist is working)

---

Code Quality Issues Found

Issue #1: Unused Function Parameter

File: core/box/free_local_box.c (line 8)

void tiny_free_local_box(SuperSlab* ss, int slab_idx, TinySlabMeta* meta, void* ptr, uint32_t my_tid) {
    // ...
    (void)my_tid;  // Explicitly ignored
}

Why: Parameter passed but not used - suggests design change where ownership was computed earlier

Issue #2: Magic Number for First Slab

File: core/hakmem_tiny_free.inc (line 676)

if (slab_idx == 0) {
    slab_start = (char*)slab_start + 1024;  // Magic number!
}

Should be:

if (slab_idx == 0) {
    slab_start = (char*)slab_start + sizeof(SuperSlab);  // or named constant
}

Issue #3: Duplicate Freelist Scan Logic

Locations:

• core/hakmem_tiny_free.inc (line ~45-62): tiny_remote_queue_contains_guard()
• core/hakmem_tiny_free.inc (line ~50-64): Duplicate in safe_free path

These should be unified into a helper function.

---

Performance Impact

Current Situation:

• Freelist is functional and pushed correctly
• But publish/fetch visibility is weak
• Forces all allocations to use direct freelist pop (bypassingrefill path)
• This is actually good for performance (fewer lock/sync operations)
• But creates hidden fragmentation (freelist not reorganized by adopt path)

After Fix:

• Expect +5-10% refill path usage (from ~0% to ~5-10%)
• Refill path can reorganize and rebalance
• Better memory locality for hot allocations
• Slightly more atomic operations during free (acceptable trade-off)

---

Conclusion

The freelist push IS happening. The bug is not in the push logic itself, but in:

1. Visibility Gap: Pushed blocks are not tracked by mailbox when accessed via direct pop
2. Incomplete Publish: Only first-free publishes; later frees are silent
3. Lack of Republish: Freelist state changes not advertised to refill path

The fixes are straightforward:

• Re-publish on every free (not just first-free)
• Validate mailbox entries during fetch
• Track direct vs. refill access to find optimal balance

This explains why Larson shows low refill metrics despite high freelist push rate.