hakmem/docs/analysis/PHASE28_BG_SPILL_ATOMIC_PRUNE_RESULTS.md

Phase 28: Background Spill Queue Atomic Prune Results

Date: 2025-12-16 Status: ✅ COMPLETE (NO-OP) Verdict: All CORRECTNESS - No compile-out candidates

---

Executive Summary

Phase 28 conducted a thorough audit of all atomic operations in the background spill queue subsystem (core/hakmem_tiny_bg_spill.*). Result: All 8 atomics are CORRECTNESS-critical. No telemetry atomics were found, therefore no compile-out was performed.

Key Finding: The g_bg_spill_len counter, which superficially resembles telemetry counters from previous phases, is actually used for flow control (queue depth limiting) and is therefore untouchable.

---

Audit Results

Total Atomics: 8

Atomic Operation	Location	Classification	Reason
atomic_load(&g_bg_spill_head)	hakmem_tiny_bg_spill.h:27	CORRECTNESS	Lock-free queue CAS loop
atomic_compare_exchange_weak(&g_bg_spill_head)	hakmem_tiny_bg_spill.h:29	CORRECTNESS	Lock-free queue CAS
atomic_fetch_add(&g_bg_spill_len, 1)	hakmem_tiny_bg_spill.h:32	CORRECTNESS	Queue length (flow control)
atomic_load(&g_bg_spill_head)	hakmem_tiny_bg_spill.h:39	CORRECTNESS	Lock-free queue CAS loop
atomic_compare_exchange_weak(&g_bg_spill_head)	hakmem_tiny_bg_spill.h:41	CORRECTNESS	Lock-free queue CAS
atomic_fetch_add(&g_bg_spill_len, count)	hakmem_tiny_bg_spill.h:44	CORRECTNESS	Queue length (flow control)
atomic_load(&g_bg_spill_len)	hakmem_tiny_bg_spill.c:30	CORRECTNESS	Early-exit optimization
atomic_fetch_sub(&g_bg_spill_len)	hakmem_tiny_bg_spill.c:91	CORRECTNESS	Queue length decrement

CORRECTNESS: 8/8 (100%) TELEMETRY: 0/8 (0%)

---

Critical Finding: g_bg_spill_len is NOT Telemetry

The Trap

At first glance, g_bg_spill_len looks like a telemetry counter:

• Named with _len suffix (like stats counters)
• Incremented on push, decremented on drain
• Uses atomic_fetch_add/sub (same pattern as telemetry)

The Reality

g_bg_spill_len is used for flow control in the hot free path:

// core/tiny_free_magazine.inc.h:75-77
if (g_bg_spill_enable) {
    uint32_t qlen = atomic_load_explicit(&g_bg_spill_len[class_idx], memory_order_relaxed);
    if ((int)qlen < g_bg_spill_target) {  // <-- FLOW CONTROL DECISION
        // Build a small chain: include current ptr and pop from mag up to limit
        // ...
        bg_spill_push_chain(class_idx, head, tail, taken);
        return;
    }
}

What this means:

• If queue length < target: queue work to background thread
• If queue length >= target: take alternate path (direct free)
• Removing this atomic would change program behavior (unbounded queue growth)
• This is an operational counter, not a debug counter

Comparison with Telemetry Counters

Counter	Phase	Purpose	Flow Control?	Classification
g_tiny_class_stats_*	24	Observe cache hits	NO	TELEMETRY
g_free_ss_enter	25	Count free calls	NO	TELEMETRY
g_unified_cache_*	27	Measure cache perf	NO	TELEMETRY
g_bg_spill_len	28	Queue depth limit	YES	CORRECTNESS

Key Distinction: Telemetry counters are observational (removed if not observed). Operational counters are functional (program behavior depends on them).

---

Lock-Free Queue Atomics

The remaining 6 atomics are part of the lock-free stack implementation:

Push Operation (lines 24-32, 36-44)

static inline void bg_spill_push_one(int class_idx, void* p) {
    uintptr_t old_head;
    do {
        old_head = atomic_load_explicit(&g_bg_spill_head[class_idx], memory_order_acquire);  // CORRECTNESS
        tiny_next_write(class_idx, p, (void*)old_head);
    } while (!atomic_compare_exchange_weak_explicit(&g_bg_spill_head[class_idx], &old_head,  // CORRECTNESS
                                                    (uintptr_t)p,
                                                    memory_order_release, memory_order_relaxed));
    atomic_fetch_add_explicit(&g_bg_spill_len[class_idx], 1u, memory_order_relaxed);  // CORRECTNESS (flow control)
}

Analysis:

• Classic lock-free stack pattern (load → link → CAS loop)
• atomic_load + atomic_compare_exchange_weak are fundamental to correctness
• Cannot be removed without replacing entire queue implementation

---

Decision: NO-OP

Verdict: Phase 28 is a NO-OP. No code changes required.

Rationale:

1. All atomics are CORRECTNESS-critical
2. g_bg_spill_len is used for flow control, not telemetry
3. Lock-free queue operations are untouchable
4. No A/B testing needed (nothing to test)

Phase 28 Result:

• Atomics removed: 0
• Performance gain: N/A
• Code changes: None
• Documentation: Audit complete, classification recorded

---

Impact on Future Phases

Lesson Learned

Not all counters are telemetry. Before classifying an atomic as TELEMETRY:

1. Search for all uses of the variable
2. Check if it's used in control flow (if, while, comparisons)
3. Determine if removal would change program behavior
4. Only compile-out if purely observational

Similar Candidates to Audit Carefully

Phase 29+ candidates that may have flow control:

• g_remote_target_len (remote queue length - same pattern as bg_spill)
• g_l25_pool.remote_count (L25 pool remote counts)
• Any *_len, *_count that might be used for queue management

Red flags for CORRECTNESS:

• Used in if (count < threshold) statements
• Used to decide whether to queue work
• Used to prevent unbounded growth
• Paired with lock-free queue head pointers

---

Phase 28 Files Analyzed

No modifications:

• core/hakmem_tiny_bg_spill.h (audit only)
• core/hakmem_tiny_bg_spill.c (audit only)
• core/tiny_free_magazine.inc.h (flow control usage identified)

Documentation created:

• docs/analysis/PHASE28_BG_SPILL_ATOMIC_AUDIT.md (detailed audit)
• docs/analysis/PHASE28_BG_SPILL_ATOMIC_PRUNE_RESULTS.md (this file)
• docs/analysis/ATOMIC_PRUNE_CUMULATIVE_SUMMARY.md (updated)

---

Cumulative Progress

Phase	Atomics Removed	Impact	Verdict
24	5	+0.93%	GO ✅
25	1	+1.07%	GO ✅
26	5	-0.33%	NEUTRAL ✅
27	6	+0.74%	GO ✅
28	0	N/A	NO-OP ✅
Total	17	+2.74%	✅

Next: Phase 29 (remote target queue or pool hotbox v2 stats)

---

Conclusion

Phase 28 successfully completed its audit objective:

1. ✅ All atomics identified (8 total)
2. ✅ All atomics classified (100% CORRECTNESS)
3. ✅ Flow control usage documented (g_bg_spill_len)
4. ✅ No compile-out candidates found
5. ✅ Cumulative summary updated

Key Takeaway: Audit phases are valuable even when they result in NO-OP. They document which atomics are untouchable and why, preventing future incorrect optimizations.

---

Phase 28 Status: ✅ COMPLETE (NO-OP) Next Phase: 29 (TBD based on priority) Date: 2025-12-16