Files
hakmem/docs/specs/ENV_VARS_COMPLETE.md
Moe Charm (CI) a9ddb52ad4 ENV cleanup: Remove BG/HotMag vars & guard fprintf (Larson 52.3M ops/s)
Phase 1 完了:環境変数整理 + fprintf デバッグガード

ENV変数削除(BG/HotMag系):
- core/hakmem_tiny_init.inc: HotMag ENV 削除 (~131 lines)
- core/hakmem_tiny_bg_spill.c: BG spill ENV 削除
- core/tiny_refill.h: BG remote 固定値化
- core/hakmem_tiny_slow.inc: BG refs 削除

fprintf Debug Guards (#if !HAKMEM_BUILD_RELEASE):
- core/hakmem_shared_pool.c: Lock stats (~18 fprintf)
- core/page_arena.c: Init/Shutdown/Stats (~27 fprintf)
- core/hakmem.c: SIGSEGV init message

ドキュメント整理:
- 328 markdown files 削除(旧レポート・重複docs)

性能確認:
- Larson: 52.35M ops/s (前回52.8M、安定動作)
- ENV整理による機能影響なし
- Debug出力は一部残存(次phase で対応)

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-26 14:45:26 +09:00

22 KiB
Raw Blame History

HAKMEM Environment Variables Complete Reference

Total Variables: 83 environment variables + multiple compile-time flags Last Updated: 2025-11-01 Purpose: Complete reference for diagnosing memory issues and configuration


CRITICAL DISCOVERY: Statistics Disabled by Default

The Problem

Tiny Pool statistics are DISABLED unless you build with -DHAKMEM_ENABLE_STATS:

  • Current behavior: alloc=0, free=0, slab=0 (statistics not collected)
  • Impact: Memory diagnostics are blind
  • Root cause: Build-time flag NOT set in Makefile

How to Enable Statistics

Option 1: Build with statistics (RECOMMENDED for debugging)

make clean
make CFLAGS="-DHAKMEM_ENABLE_STATS" bench_fragment_stress_hakmem

Option 2: Edit Makefile (add to line 18)

CFLAGS = -O3 ... -DHAKMEM_ENABLE_STATS ...

Why Statistics are Disabled

From /mnt/workdisk/public_share/hakmem/core/hakmem_tiny_stats.h:

// Purpose: Zero-overhead production builds by disabling stats collection
// Usage:   Build with -DHAKMEM_ENABLE_STATS to enable (default: disabled)
// Impact:  3-5% speedup when disabled (removes 0.5ns TLS increment)
//
// Default: DISABLED (production performance)
// Enable:  make CFLAGS=-DHAKMEM_ENABLE_STATS

When DISABLED: All stats_record_alloc() and stats_record_free() become no-ops When ENABLED: Batched TLS counters track exact allocation/free counts


Environment Variable Categories

1. Tiny Pool Core (Critical)

HAKMEM_WRAP_TINY

  • Default: 1 (enabled)
  • Purpose: Enable Tiny Pool fast-path (bypasses wrapper guard)
  • Impact: Controls whether malloc/free use Tiny Pool for ≤1KB allocations
  • Usage: export HAKMEM_WRAP_TINY=1 (already default since Phase 7.4)
  • Location: /mnt/workdisk/public_share/hakmem/core/hakmem_tiny_init.inc:25
  • Notes: Without this, Tiny Pool returns NULL and falls back to L2/L25

HAKMEM_WRAP_TINY_REFILL

  • Default: 0 (disabled)
  • Purpose: Allow trylock-based magazine refill during wrapper calls
  • Impact: Enables limited refill under trylock (no blocking)
  • Usage: export HAKMEM_WRAP_TINY_REFILL=1
  • Safety: OFF by default (avoids deadlock risk in recursive malloc)

HAKMEM_TINY_USE_SUPERSLAB

  • Default: 1 (enabled)
  • Purpose: Enable SuperSlab allocator for Tiny Pool slabs
  • Impact: When OFF, Tiny Pool cannot allocate new slabs
  • Critical: Must be ON for Tiny Pool to work

2. Tiny Pool TLS Caching (Performance Critical)

HAKMEM_TINY_MAG_CAP

  • Default: Per-class (typically 512-2048)
  • Purpose: Global TLS magazine capacity override
  • Impact: Larger = fewer refills, more memory
  • Usage: export HAKMEM_TINY_MAG_CAP=1024

HAKMEM_TINY_MAG_CAP_C{0..7}

  • Default: None (uses class defaults)
  • Purpose: Per-class magazine capacity override
  • Example: HAKMEM_TINY_MAG_CAP_C3=512 (64B class)
  • Classes: C0=8B, C1=16B, C2=32B, C3=64B, C4=128B, C5=256B, C6=512B, C7=1KB

HAKMEM_TINY_TLS_SLL

  • Default: 1 (enabled)
  • Purpose: Enable TLS Single-Linked-List cache layer
  • Impact: Fast-path cache before magazine
  • Performance: Critical for tiny allocations (8-64B)

HAKMEM_SLL_MULTIPLIER

  • Default: 2
  • Purpose: SLL capacity = MAG_CAP × multiplier for small classes (0-3)
  • Range: 1..16
  • Impact: Higher = more TLS memory, fewer refills

HAKMEM_TINY_REFILL_MAX

  • Default: 64
  • Purpose: Magazine refill batch size (normal classes)
  • Impact: Larger = fewer refills, more memory spike

HAKMEM_TINY_REFILL_MAX_HOT

  • Default: 192
  • Purpose: Magazine refill batch size for hot classes (≤64B)
  • Impact: Larger batches for frequently used sizes

HAKMEM_TINY_REFILL_MAX_C{0..7}

  • Default: None
  • Purpose: Per-class refill batch override
  • Example: HAKMEM_TINY_REFILL_MAX_C2=96 (32B class)

HAKMEM_TINY_REFILL_MAX_HOT_C{0..7}

  • Default: None
  • Purpose: Per-class hot refill override (classes 0-3)
  • Priority: Overrides HAKMEM_TINY_REFILL_MAX_HOT

3. SuperSlab Configuration

HAKMEM_TINY_SS_MAX_MB

  • Default: Unlimited
  • Purpose: Maximum SuperSlab memory per class (MB)
  • Impact: Caps total slab allocation
  • Usage: export HAKMEM_TINY_SS_MAX_MB=512

HAKMEM_TINY_SS_MIN_MB

  • Default: 0
  • Purpose: Minimum SuperSlab reservation per class (MB)
  • Impact: Pre-allocates memory at startup

HAKMEM_TINY_SS_RESERVE

  • Default: 0
  • Purpose: Reserve SuperSlab memory at init
  • Impact: Prevents initial allocation delays

HAKMEM_TINY_TRIM_SS

  • Default: 0
  • Purpose: Enable SuperSlab trimming/deallocation
  • Impact: Returns memory to OS when idle

HAKMEM_TINY_SS_PARTIAL

  • Default: 0
  • Purpose: Enable partial slab reclamation
  • Impact: Free partially-used slabs

HAKMEM_TINY_SS_PARTIAL_INTERVAL

  • Default: 1000000 (1M allocations)
  • Purpose: Interval between partial slab checks
  • Impact: Lower = more aggressive trimming

4. Remote Free & Background Processing

HAKMEM_TINY_REMOTE_DRAIN_THRESHOLD

  • Default: 32
  • Purpose: Trigger remote free drain when count exceeds threshold
  • Impact: Controls when to process cross-thread frees
  • Per-class: ACE can tune this per-class

HAKMEM_TINY_REMOTE_DRAIN_TRYRATE

  • Default: 16
  • Purpose: Probability (1/N) of attempting trylock drain
  • Impact: Lower = more aggressive draining

HAKMEM_TINY_BG_REMOTE (削除済み)

  • 2025-12 cleanup: BG Remote系ENVは廃止。BGリモートドレインは固定OFF。

HAKMEM_TINY_BG_REMOTE_BATCH (削除済み)

  • 2025-12 cleanup: BG Remote batch ENVは廃止固定値32未使用

HAKMEM_TINY_BG_SPILL (削除済み)

  • 2025-12 cleanup: BG Spill系ENVは廃止。BG spillは固定OFF。

HAKMEM_TINY_BG_BIN / HAKMEM_TINY_BG_TARGET (削除済み)

  • 2025-12 cleanup: BG Bin/Target ENVは廃止BG bin処理は固定OFF

5. Statistics & Profiling

HAKMEM_ENABLE_STATS (BUILD-TIME)

  • Default: UNDEFINED (statistics DISABLED)
  • Purpose: Enable batched TLS statistics collection
  • Build: make CFLAGS=-DHAKMEM_ENABLE_STATS
  • Impact: 0.5ns overhead per alloc/free when enabled
  • Critical: Must be defined to see any statistics

HAKMEM_TINY_STAT_RATE_LG

  • Default: 0 (no sampling)
  • Purpose: Sample statistics at 1/2^N rate
  • Example: HAKMEM_TINY_STAT_RATE_LG=4 → sample 1/16 allocs
  • Requires: HAKMEM_ENABLE_STATS + HAKMEM_TINY_STAT_SAMPLING build flags

HAKMEM_TINY_COUNT_SAMPLE

  • Default: 8
  • Purpose: Legacy sampling exponent (deprecated)
  • Note: Replaced by batched stats in Phase 3

HAKMEM_TINY_PATH_DEBUG

  • Default: 0
  • Purpose: Enable allocation path debugging counters
  • Requires: HAKMEM_DEBUG_COUNTERS=1 build flag
  • Output: atexit() dump of path hit counts

6. ACE Learning System (Adaptive Control Engine)

HAKMEM_ACE_ENABLED

  • Default: 0
  • Purpose: Enable ACE learning system
  • Impact: Adaptive tuning of Tiny Pool parameters
  • Note: Already integrated but can be disabled

HAKMEM_ACE_OBSERVE

  • Default: 0
  • Purpose: Enable ACE observation logging
  • Impact: Verbose output of ACE decisions

HAKMEM_ACE_DEBUG

  • Default: 0
  • Purpose: Enable ACE debug logging
  • Impact: Detailed ACE internal state

HAKMEM_ACE_SAMPLE

  • Default: Undefined (no sampling)
  • Purpose: Sample ACE events at given rate
  • Impact: Reduces ACE overhead

HAKMEM_ACE_LOG_LEVEL

  • Default: 0
  • Purpose: ACE logging verbosity (0-3)
  • Levels: 0=off, 1=errors, 2=info, 3=debug

HAKMEM_ACE_FAST_INTERVAL_MS

  • Default: 100ms
  • Purpose: Fast ACE update interval
  • Impact: How often ACE checks metrics

HAKMEM_ACE_SLOW_INTERVAL_MS

  • Default: 1000ms
  • Purpose: Slow ACE update interval
  • Impact: Background tuning frequency

7. Intelligence Engine (INT)

HAKMEM_INT_ENGINE

  • Default: 0
  • Purpose: Enable background intelligence/adaptation engine
  • Impact: Deferred event processing + adaptive tuning
  • Pairs with: HAKMEM_TINY_FRONTEND

HAKMEM_INT_ADAPT_REFILL

  • Default: 1 (when INT enabled)
  • Purpose: Adapt REFILL_MAX dynamically (±16)
  • Impact: Tunes refill sizes based on miss rate

HAKMEM_INT_ADAPT_CAPS

  • Default: 1 (when INT enabled)
  • Purpose: Adapt MAG/SLL capacities (±16/±32)
  • Impact: Grows hot classes, shrinks cold ones

HAKMEM_INT_EVENT_TS

  • Default: 0
  • Purpose: Include timestamps in INT events
  • Impact: Adds clock_gettime() overhead

HAKMEM_INT_SAMPLE

  • Default: Undefined (no sampling)
  • Purpose: Sample INT events at 1/2^N rate
  • Impact: Reduces INT overhead on hot path

8. Frontend & Experimental Features

HAKMEM_TINY_FRONTEND

  • Default: 0
  • Purpose: Enable mimalloc-style frontend cache
  • Impact: Adds FastCache layer before backend
  • Experimental: A/B testing only

HAKMEM_TINY_FASTCACHE

  • Default: 0
  • Purpose: Low-level FastCache toggle
  • Impact: Internal A/B switch

HAKMEM_TINY_QUICK

  • Default: 0
  • Purpose: Enable TinyQuickSlot (6-item single-cacheline stack)
  • Impact: Ultra-fast path for ≤64B
  • Experimental: Bench-only optimization

HAKMEM_TINY_HOTMAG (削除済み)

  • 2025-12 cleanup: HotMag runtime ENVトグルは削除。HotMagはデフォルトOFF固定、ENVでの調整不可。

HAKMEM_TINY_HOTMAG_CAP (削除済み)

  • 2025-12 cleanup: HotMag容量ENVを削除固定値128

HAKMEM_TINY_HOTMAG_REFILL (削除済み)

  • 2025-12 cleanup: HotMag refillバッチENVを削除固定値32

HAKMEM_TINY_HOTMAG_C{0..7} (削除済み)

  • 2025-12 cleanup: クラス別HotMag有効/無効ENVを削除全クラス固定OFF

9. Memory Efficiency & RSS Control

HAKMEM_TINY_RSS_BUDGET_KB

  • Default: Unlimited
  • Purpose: Total RSS budget for Tiny Pool (kB)
  • Impact: When exceeded, shrinks MAG/SLL capacities
  • INT interaction: Requires HAKMEM_INT_ENGINE=1

HAKMEM_TINY_INT_TIGHT

  • Default: 0
  • Purpose: Bias INT toward memory reduction
  • Impact: Higher shrink thresholds, lower floor values

HAKMEM_TINY_DIET_STEP

  • Default: 16
  • Purpose: Capacity reduction step when over budget
  • Impact: MAG -= step, SLL -= step×2

HAKMEM_TINY_CAP_FLOOR_C{0..7}

  • Default: None (no floor)
  • Purpose: Minimum MAG capacity per class
  • Example: HAKMEM_TINY_CAP_FLOOR_C0=64 (8B class min)
  • Impact: Prevents INT from shrinking below floor

HAKMEM_TINY_MEM_DIET

  • Default: 0
  • Purpose: Enable memory diet mode (aggressive trimming)
  • Impact: Reduces memory footprint at cost of performance

HAKMEM_TINY_SPILL_HYST

  • Default: 0
  • Purpose: Magazine spill hysteresis (avoid thrashing)
  • Impact: Keep N extra items before spilling

10. Policy & Learning Parameters

HAKMEM_LEARN

  • Default: 0
  • Purpose: Enable global learning mode
  • Impact: Activates UCB1/ELO/THP learning

HAKMEM_WMAX_MID

  • Default: 256KB
  • Purpose: Mid-size allocation working set max
  • Impact: Pool cache size for mid-tier

HAKMEM_WMAX_LARGE

  • Default: 2MB
  • Purpose: Large allocation working set max
  • Impact: Pool cache size for large-tier

HAKMEM_CAP_MID

  • Default: Unlimited
  • Purpose: Mid-tier pool capacity cap
  • Impact: Maximum mid-tier pool size

HAKMEM_CAP_LARGE

  • Default: Unlimited
  • Purpose: Large-tier pool capacity cap
  • Impact: Maximum large-tier pool size

HAKMEM_WMAX_LEARN

  • Default: 0
  • Purpose: Enable working set max learning
  • Impact: Adaptively tune WMAX based on hit rate

HAKMEM_WMAX_CANDIDATES_MID

  • Default: "128,256,512,1024"
  • Purpose: Candidate WMAX values for mid-tier learning
  • Format: Comma-separated KB values

HAKMEM_WMAX_CANDIDATES_LARGE

  • Default: "1024,2048,4096,8192"
  • Purpose: Candidate WMAX values for large-tier learning
  • Format: Comma-separated KB values

HAKMEM_WMAX_ADOPT_PCT

  • Default: 0.01 (1%)
  • Purpose: Adoption threshold for WMAX candidates
  • Impact: How much better to switch candidates

HAKMEM_TARGET_HIT_MID

  • Default: 0.65 (65%)
  • Purpose: Target hit rate for mid-tier
  • Impact: Learning objective

HAKMEM_TARGET_HIT_LARGE

  • Default: 0.55 (55%)
  • Purpose: Target hit rate for large-tier
  • Impact: Learning objective

HAKMEM_GAIN_W_MISS

  • Default: 1.0
  • Purpose: Learning gain weight for misses
  • Impact: How much to penalize misses

11. THP (Transparent Huge Pages)

HAKMEM_THP

  • Default: "auto"
  • Purpose: THP policy (off/auto/on)
  • Values:
    • "off" = MADV_NOHUGEPAGE for all
    • "auto" = ≥2MB → MADV_HUGEPAGE
    • "on" = MADV_HUGEPAGE for all ≥1MB

HAKMEM_THP_LEARN

  • Default: 0
  • Purpose: Enable THP policy learning
  • Impact: Adaptively choose THP policy

HAKMEM_THP_CANDIDATES

  • Default: "off,auto,on"
  • Purpose: THP candidate policies for learning
  • Format: Comma-separated

HAKMEM_THP_ADOPT_PCT

  • Default: 0.015 (1.5%)
  • Purpose: Adoption threshold for THP switch
  • Impact: How much better to switch

12. L2/L25 Pool Configuration

HAKMEM_WRAP_L2

  • Default: 0
  • Purpose: Enable L2 pool wrapper bypass
  • Impact: Allow L2 during wrapper calls

HAKMEM_WRAP_L25

  • Default: 0
  • Purpose: Enable L25 pool wrapper bypass
  • Impact: Allow L25 during wrapper calls

HAKMEM_POOL_TLS_FREE

  • Default: 1
  • Purpose: Enable TLS-local free for L2 pool
  • Impact: Lock-free fast path

HAKMEM_POOL_TLS_RING

  • Default: 1
  • Purpose: Enable TLS ring buffer for pool
  • Impact: Batched cross-thread returns

HAKMEM_POOL_MIN_BUNDLE

  • Default: 4
  • Purpose: Minimum bundle size for L2 pool
  • Impact: Batch refill size

HAKMEM_L25_MIN_BUNDLE

  • Default: 4
  • Purpose: Minimum bundle size for L25 pool
  • Impact: Batch refill size

HAKMEM_L25_DZ

  • Default: "64,256"
  • Purpose: L25 size zones (comma-separated)
  • Format: "size1,size2,..."

HAKMEM_L25_RUN_BLOCKS

  • Default: 16
  • Purpose: Run blocks per L25 slab
  • Impact: Slab structure

HAKMEM_L25_RUN_FACTOR

  • Default: 2
  • Purpose: Run factor multiplier
  • Impact: Slab allocation strategy

13. Debugging & Observability

HAKMEM_VERBOSE

  • Default: 0
  • Purpose: Enable verbose logging
  • Impact: Detailed allocation logs

HAKMEM_QUIET

  • Default: 0
  • Purpose: Suppress all logging
  • Impact: Overrides HAKMEM_VERBOSE

HAKMEM_TIMING

  • Default: 0
  • Purpose: Enable timing measurements
  • Impact: Track allocation latency

HAKMEM_HIST_SAMPLE

  • Default: 0
  • Purpose: Size histogram sampling rate
  • Impact: Track size distribution

HAKMEM_PROF

  • Default: 0
  • Purpose: Enable profiling mode
  • Impact: Detailed performance tracking

HAKMEM_LOG_FILE

  • Default: stderr
  • Purpose: Redirect logs to file
  • Impact: File path for logging output

14. Mode Presets

HAKMEM_MODE

  • Default: "balanced"
  • Purpose: High-level configuration preset
  • Values:
    • "minimal" = malloc/mmap only
    • "fast" = pool fast-path + frozen learning
    • "balanced" = BigCache + ELO + Batch (default)
    • "learning" = ELO LEARN + adaptive
    • "research" = all features + verbose

HAKMEM_PRESET

  • Default: None
  • Purpose: Evolution preset (from PRESETS.md)
  • Impact: Load predefined parameter set

HAKMEM_FREE_POLICY

  • Default: "batch"
  • Purpose: Free path policy
  • Values: "batch", "keep", "adaptive"

15. Build-Time Flags (Not Environment Variables)

HAKMEM_ENABLE_STATS

  • Type: Compiler flag (-DHAKMEM_ENABLE_STATS)
  • Default: NOT DEFINED
  • Impact: Completely disables statistics when absent
  • Critical: Must be set to collect any statistics

HAKMEM_BUILD_RELEASE

  • Type: Compiler flag
  • Default: NOT DEFINED (= 0)
  • Impact: When undefined, enables debug paths
  • Check: #if !HAKMEM_BUILD_RELEASE = true when not set

HAKMEM_BUILD_DEBUG

  • Type: Compiler flag
  • Default: NOT DEFINED (= 0)
  • Impact: Enables debug counters and logging

HAKMEM_DEBUG_COUNTERS

  • Type: Compiler flag
  • Default: 0
  • Impact: Include path debug counters in build

HAKMEM_TINY_MINIMAL_FRONT

  • Type: Compiler flag
  • Default: 0
  • Impact: Strip optional front-end layers (bench only)

HAKMEM_TINY_BENCH_FASTPATH

  • Type: Compiler flag
  • Default: 0
  • Impact: Enable benchmark-optimized fast path

HAKMEM_TINY_BENCH_SLL_ONLY

  • Type: Compiler flag
  • Default: 0
  • Impact: SLL-only mode (no magazines)

HAKMEM_USDT

  • Type: Compiler flag
  • Default: 0
  • Impact: Enable USDT tracepoints for perf
  • Requires: <sys/sdt.h> (systemtap-sdt-dev)

NULL Return Path Analysis

Why hak_tiny_alloc() Returns NULL

The Tiny Pool allocator returns NULL in these cases:

  1. Size > 1KB (line 97)

    if (class_idx < 0) return NULL;  // >1KB
    
  2. Wrapper Guard Active (lines 88-91, only when !HAKMEM_BUILD_RELEASE)

    #if !HAKMEM_BUILD_RELEASE
    if (!g_wrap_tiny_enabled && g_tls_in_wrapper != 0) return NULL;
    #endif
    

    Note: HAKMEM_BUILD_RELEASE is NOT defined by default! This guard is ACTIVE in your build and returns NULL during malloc recursion.

  3. Wrapper Context Empty (line 73)

    return NULL;  // empty → fallback to next allocator tier
    

    Called from hak_tiny_alloc_wrapper() when magazine is empty.

  4. Slow Path Exhaustion When all of these fail in hak_tiny_alloc_slow():

    • HotMag refill fails
    • TLS list empty
    • TLS slab refill fails
    • hak_tiny_alloc_superslab() returns NULL

When Tiny Pool is Bypassed

Given HAKMEM_WRAP_TINY=1 (default), Tiny Pool is still bypassed when:

  1. During wrapper recursion (if HAKMEM_BUILD_RELEASE not set)

    • malloc() calls getenv()
    • getenv() calls malloc()
    • Guard returns NULL → falls back to L2/L25
  2. Size > 1KB

    • Always falls through to L2 pool (1KB-32KB)
  3. All caches empty + SuperSlab allocation fails

    • Magazine empty
    • SLL empty
    • Active slabs full
    • SuperSlab cannot allocate new slab
    • Falls back to L2/L25

Memory Issue Diagnosis: 9GB Usage

Current Symptoms

  • bench_fragment_stress_long_hakmem: 9GB RSS
  • System allocator: 1.6MB RSS
  • Tiny Pool stats: alloc=0, free=0, slab=0 (ZERO activity)

Root Cause Analysis

Hypothesis #1: Statistics Disabled (CONFIRMED)

Probability: 100%

Evidence:

  • HAKMEM_ENABLE_STATS not defined in Makefile
  • All stats show 0 (no data collection)
  • Code in hakmem_tiny_stats.h:243-275 shows no-op when disabled

Impact:

  • Cannot see if Tiny Pool is being used
  • Cannot diagnose allocation patterns
  • Blind to memory leaks

Fix:

make clean
make CFLAGS="-DHAKMEM_ENABLE_STATS" bench_fragment_stress_hakmem

Hypothesis #2: Wrapper Guard Blocking Tiny Pool

Probability: 90%

Evidence:

  • HAKMEM_BUILD_RELEASE not defined → guard is ACTIVE
  • Wrapper guard code at hakmem_tiny_alloc.inc:86-92
  • During benchmark, many allocations may trigger wrapper context

Mechanism:

#if !HAKMEM_BUILD_RELEASE  // This is TRUE (not defined)
if (!g_wrap_tiny_enabled && g_tls_in_wrapper != 0)
    return NULL;  // Bypass Tiny Pool!
#endif

Result:

  • Tiny Pool returns NULL
  • Falls back to L2/L25 pools
  • L2/L25 may be leaking or over-allocating

Fix:

make CFLAGS="-DHAKMEM_BUILD_RELEASE=1"

Hypothesis #3: L2/L25 Pool Leak or Over-Retention

Probability: 75%

Evidence:

  • If Tiny Pool is bypassed → L2/L25 handles ≤1KB allocations
  • L2/L25 may have less aggressive trimming
  • Fragment stress workload may trigger worst-case pooling

Verification:

  1. Enable L2/L25 statistics
  2. Check pool sizes: g_pool_* counters
  3. Look for unbounded pool growth

Fix: Tune L2/L25 parameters:

export HAKMEM_POOL_TLS_FREE=1
export HAKMEM_CAP_MID=256  # Cap mid-tier pool at 256 blocks

Step 1: Enable Statistics

make clean
make CFLAGS="-DHAKMEM_ENABLE_STATS -DHAKMEM_BUILD_RELEASE=1" bench_fragment_stress_hakmem

Step 2: Run with Diagnostics

export HAKMEM_WRAP_TINY=1
export HAKMEM_VERBOSE=1
./bench_fragment_stress_hakmem

Step 3: Check Statistics

# In benchmark output, look for:
# - Tiny Pool stats (should be non-zero now)
# - L2/L25 pool stats
# - Cache hit rates
# - RSS growth pattern

Step 4: Profile Memory

# Option A: Valgrind massif
valgrind --tool=massif --massif-out-file=massif.out ./bench_fragment_stress_hakmem
ms_print massif.out

# Option B: HAKMEM internal profiling
export HAKMEM_PROF=1
export HAKMEM_PROF_SAMPLE=100
./bench_fragment_stress_hakmem

Step 5: Compare Allocator Tiers

# Force Tiny-only (disable L2/L25 fallback)
export HAKMEM_TINY_USE_SUPERSLAB=1
export HAKMEM_CAP_MID=0      # Disable mid-tier
export HAKMEM_CAP_LARGE=0    # Disable large-tier
./bench_fragment_stress_hakmem

# Check if RSS improves → L2/L25 is the problem

Quick Reference: Must-Set Variables for Debugging

# Enable everything for debugging
export HAKMEM_WRAP_TINY=1              # Use Tiny Pool
export HAKMEM_VERBOSE=1                # See what's happening
export HAKMEM_ACE_DEBUG=1              # ACE diagnostics
export HAKMEM_TINY_PATH_DEBUG=1        # Path counters (if built with HAKMEM_DEBUG_COUNTERS)

# Build with statistics
make clean
make CFLAGS="-DHAKMEM_ENABLE_STATS -DHAKMEM_BUILD_RELEASE=1 -DHAKMEM_DEBUG_COUNTERS=1"

Summary: Critical Variables for Your Issue

Variable Current Should Be Impact
HAKMEM_ENABLE_STATS undefined -DHAKMEM_ENABLE_STATS Enable statistics collection
HAKMEM_BUILD_RELEASE undefined (=0) -DHAKMEM_BUILD_RELEASE=1 Disable wrapper guard
HAKMEM_WRAP_TINY 1 ✓ 1 Already correct
HAKMEM_VERBOSE 0 1 See allocation logs

Action: Rebuild with both flags, then re-run benchmark to see real statistics.