Phase 21-1-A: Ring cache 基本実装 - Array-based TLS cache (C2/C3)
## Summary Phase 21-1-A の基本実装完了。Ring buffer ベースの TLS cache を C2/C3 （33-128B）専用に実装。ポインタチェイス削減で +15-20% 性能向上を目指す。 ## Implementation **Files Created**: - `core/front/tiny_ring_cache.h` - Ring cache API, ENV control - `core/front/tiny_ring_cache.c` - Ring cache implementation **Makefile Integration**: - Added `core/front/tiny_ring_cache.o` to OBJS_BASE - Added `core/front/tiny_ring_cache_shared.o` to SHARED_OBJS - Added `core/front/tiny_ring_cache.o` to BENCH_HAKMEM_OBJS_BASE ## Design (Task 先生調査結果 + ChatGPT フィードバック) **Ring Buffer Structure**: - C2/C3 専用（hot classes, 33-128B） - Default 128 slots (power-of-2, ENV で 64/128/256 A/B 可能) - Ultra-fast pop/push (1-2 instructions, array access) - Fast modulo via mask (capacity - 1) **Hierarchy** (Option 4: UltraHot 置き換え): ``` Ring (L0, C2/C3 専用) → HeapV2 (L1, fallback) → TLS SLL (L2) → SuperSlab (L3) ``` **Rationale**: - UltraHot の C3 問題（5.8% hit rate）を根本解決 - Phase 19-3 の +12.9%（UltraHot 除去）を維持 - Ring サイズ（128）>> UltraHot（4）→ hit rate 大幅向上期待 **Performance Goal**: - Pointer chasing: TLS SLL 1 回 → Ring 0 回 - Memory access: 3 → 2 回 - Cache locality: 配列（連続メモリ）vs linked list - Expected: +15-20% (54.4M → 62-65M ops/s) ## ENV Variables ```bash HAKMEM_TINY_HOT_RING_ENABLE=1 # Ring 有効化 (default: 0) HAKMEM_TINY_HOT_RING_C2=128 # C2 サイズ (default: 128) HAKMEM_TINY_HOT_RING_C3=128 # C3 サイズ (default: 128) HAKMEM_TINY_HOT_RING_CASCADE=1 # SLL → Ring refill (default: 0) ``` ## Implementation Status Phase 21-1-A: ✅ **COMPLETE** - Ring buffer data structure - TLS variables - ENV control (enable/capacity) - Power-of-2 capacity (fast modulo) - Ultra-fast pop/push inline functions - Refill from SLL (scaffold) - Init/shutdown/stats (scaffold) - Makefile integration - Compile success Phase 21-1-B: ⏳ **NEXT** - Alloc/Free 統合 Phase 21-1-C: ⏳ **PENDING** - Refill/Cascade 実装 Phase 21-1-D: ⏳ **PENDING** - A/B テスト ## Next Steps 1. Alloc path 統合 (`core/tiny_alloc_fast.inc.h`) 2. Free path 統合 (`core/tiny_free_fast_v2.inc.h`) 3. Init call from `hakmem_tiny.c` 4. A/B test: Ring vs UltraHot vs Baseline 🎯 Target: 62-65M ops/s (+15-20% vs 54.4M baseline) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Moe Charm (CI)
6
Makefile
6
Makefile
@ -190,12 +190,12 @@ LDFLAGS += $(EXTRA_LDFLAGS)
|
||||
|
||||
# Targets
|
||||
TARGET = test_hakmem
|
||||
OBJS_BASE = hakmem.o hakmem_config.o hakmem_tiny_config.o hakmem_ucb1.o hakmem_bigcache.o hakmem_pool.o hakmem_l25_pool.o hakmem_site_rules.o hakmem_tiny.o hakmem_tiny_superslab.o hakmem_smallmid.o hakmem_smallmid_superslab.o tiny_sticky.o tiny_remote.o tiny_publish.o tiny_debug_ring.o hakmem_tiny_magazine.o hakmem_tiny_stats.o hakmem_tiny_sfc.o hakmem_tiny_query.o hakmem_tiny_rss.o hakmem_tiny_registry.o hakmem_tiny_remote_target.o hakmem_tiny_bg_spill.o tiny_adaptive_sizing.o hakmem_mid_mt.o hakmem_super_registry.o hakmem_shared_pool.o hakmem_elo.o hakmem_batch.o hakmem_p2.o hakmem_sizeclass_dist.o hakmem_evo.o hakmem_debug.o hakmem_sys.o hakmem_whale.o hakmem_policy.o hakmem_ace.o hakmem_ace_stats.o hakmem_prof.o hakmem_learner.o hakmem_size_hist.o hakmem_learn_log.o hakmem_syscall.o hakmem_ace_metrics.o hakmem_ace_ucb1.o hakmem_ace_controller.o tiny_fastcache.o core/box/superslab_expansion_box.o core/box/integrity_box.o core/box/free_local_box.o core/box/free_remote_box.o core/box/free_publish_box.o core/box/mailbox_box.o core/box/front_gate_box.o core/box/front_gate_classifier.o core/box/capacity_box.o core/box/carve_push_box.o core/box/prewarm_box.o core/box/ss_hot_prewarm_box.o core/box/front_metrics_box.o core/box/bench_fast_box.o core/link_stubs.o core/tiny_failfast.o test_hakmem.o
|
||||
OBJS_BASE = hakmem.o hakmem_config.o hakmem_tiny_config.o hakmem_ucb1.o hakmem_bigcache.o hakmem_pool.o hakmem_l25_pool.o hakmem_site_rules.o hakmem_tiny.o hakmem_tiny_superslab.o hakmem_smallmid.o hakmem_smallmid_superslab.o tiny_sticky.o tiny_remote.o tiny_publish.o tiny_debug_ring.o hakmem_tiny_magazine.o hakmem_tiny_stats.o hakmem_tiny_sfc.o hakmem_tiny_query.o hakmem_tiny_rss.o hakmem_tiny_registry.o hakmem_tiny_remote_target.o hakmem_tiny_bg_spill.o tiny_adaptive_sizing.o hakmem_mid_mt.o hakmem_super_registry.o hakmem_shared_pool.o hakmem_elo.o hakmem_batch.o hakmem_p2.o hakmem_sizeclass_dist.o hakmem_evo.o hakmem_debug.o hakmem_sys.o hakmem_whale.o hakmem_policy.o hakmem_ace.o hakmem_ace_stats.o hakmem_prof.o hakmem_learner.o hakmem_size_hist.o hakmem_learn_log.o hakmem_syscall.o hakmem_ace_metrics.o hakmem_ace_ucb1.o hakmem_ace_controller.o tiny_fastcache.o core/box/superslab_expansion_box.o core/box/integrity_box.o core/box/free_local_box.o core/box/free_remote_box.o core/box/free_publish_box.o core/box/mailbox_box.o core/box/front_gate_box.o core/box/front_gate_classifier.o core/box/capacity_box.o core/box/carve_push_box.o core/box/prewarm_box.o core/box/ss_hot_prewarm_box.o core/box/front_metrics_box.o core/box/bench_fast_box.o core/front/tiny_ring_cache.o core/link_stubs.o core/tiny_failfast.o test_hakmem.o
|
||||
OBJS = $(OBJS_BASE)
|
||||
|
||||
# Shared library
|
||||
SHARED_LIB = libhakmem.so
|
||||
SHARED_OBJS = hakmem_shared.o hakmem_config_shared.o hakmem_tiny_config_shared.o hakmem_ucb1_shared.o hakmem_bigcache_shared.o hakmem_pool_shared.o hakmem_l25_pool_shared.o hakmem_site_rules_shared.o hakmem_tiny_shared.o hakmem_tiny_superslab_shared.o hakmem_smallmid_shared.o core/box/superslab_expansion_box_shared.o core/box/integrity_box_shared.o core/box/mailbox_box_shared.o core/box/front_gate_box_shared.o core/box/free_local_box_shared.o core/box/free_remote_box_shared.o core/box/free_publish_box_shared.o core/box/capacity_box_shared.o core/box/carve_push_box_shared.o core/box/prewarm_box_shared.o core/box/bench_fast_box_shared.o tiny_sticky_shared.o tiny_remote_shared.o tiny_publish_shared.o tiny_debug_ring_shared.o hakmem_tiny_magazine_shared.o hakmem_tiny_stats_shared.o hakmem_tiny_sfc_shared.o hakmem_tiny_query_shared.o hakmem_tiny_rss_shared.o hakmem_tiny_registry_shared.o hakmem_tiny_remote_target_shared.o hakmem_tiny_bg_spill_shared.o tiny_adaptive_sizing_shared.o hakmem_mid_mt_shared.o hakmem_super_registry_shared.o hakmem_elo_shared.o hakmem_batch_shared.o hakmem_p2_shared.o hakmem_sizeclass_dist_shared.o hakmem_evo_shared.o hakmem_debug_shared.o hakmem_sys_shared.o hakmem_whale_shared.o hakmem_policy_shared.o hakmem_ace_shared.o hakmem_ace_stats_shared.o hakmem_ace_controller_shared.o hakmem_ace_metrics_shared.o hakmem_ace_ucb1_shared.o hakmem_prof_shared.o hakmem_learner_shared.o hakmem_size_hist_shared.o hakmem_learn_log_shared.o hakmem_syscall_shared.o tiny_fastcache_shared.o
|
||||
SHARED_OBJS = hakmem_shared.o hakmem_config_shared.o hakmem_tiny_config_shared.o hakmem_ucb1_shared.o hakmem_bigcache_shared.o hakmem_pool_shared.o hakmem_l25_pool_shared.o hakmem_site_rules_shared.o hakmem_tiny_shared.o hakmem_tiny_superslab_shared.o hakmem_smallmid_shared.o core/box/superslab_expansion_box_shared.o core/box/integrity_box_shared.o core/box/mailbox_box_shared.o core/box/front_gate_box_shared.o core/box/free_local_box_shared.o core/box/free_remote_box_shared.o core/box/free_publish_box_shared.o core/box/capacity_box_shared.o core/box/carve_push_box_shared.o core/box/prewarm_box_shared.o core/box/bench_fast_box_shared.o core/front/tiny_ring_cache_shared.o tiny_sticky_shared.o tiny_remote_shared.o tiny_publish_shared.o tiny_debug_ring_shared.o hakmem_tiny_magazine_shared.o hakmem_tiny_stats_shared.o hakmem_tiny_sfc_shared.o hakmem_tiny_query_shared.o hakmem_tiny_rss_shared.o hakmem_tiny_registry_shared.o hakmem_tiny_remote_target_shared.o hakmem_tiny_bg_spill_shared.o tiny_adaptive_sizing_shared.o hakmem_mid_mt_shared.o hakmem_super_registry_shared.o hakmem_elo_shared.o hakmem_batch_shared.o hakmem_p2_shared.o hakmem_sizeclass_dist_shared.o hakmem_evo_shared.o hakmem_debug_shared.o hakmem_sys_shared.o hakmem_whale_shared.o hakmem_policy_shared.o hakmem_ace_shared.o hakmem_ace_stats_shared.o hakmem_ace_controller_shared.o hakmem_ace_metrics_shared.o hakmem_ace_ucb1_shared.o hakmem_prof_shared.o hakmem_learner_shared.o hakmem_size_hist_shared.o hakmem_learn_log_shared.o hakmem_syscall_shared.o tiny_fastcache_shared.o
|
||||
|
||||
# Pool TLS Phase 1 (enable with POOL_TLS_PHASE1=1)
|
||||
ifeq ($(POOL_TLS_PHASE1),1)
|
||||
@ -222,7 +222,7 @@ endif
|
||||
# Benchmark targets
|
||||
BENCH_HAKMEM = bench_allocators_hakmem
|
||||
BENCH_SYSTEM = bench_allocators_system
|
||||
BENCH_HAKMEM_OBJS_BASE = hakmem.o hakmem_config.o hakmem_tiny_config.o hakmem_ucb1.o hakmem_bigcache.o hakmem_pool.o hakmem_l25_pool.o hakmem_site_rules.o hakmem_tiny.o hakmem_tiny_superslab.o hakmem_smallmid.o tiny_sticky.o tiny_remote.o tiny_publish.o tiny_debug_ring.o hakmem_tiny_magazine.o hakmem_tiny_stats.o hakmem_tiny_sfc.o hakmem_tiny_query.o hakmem_tiny_rss.o hakmem_tiny_registry.o hakmem_tiny_remote_target.o hakmem_tiny_bg_spill.o tiny_adaptive_sizing.o hakmem_mid_mt.o hakmem_super_registry.o hakmem_elo.o hakmem_batch.o hakmem_p2.o hakmem_sizeclass_dist.o hakmem_evo.o hakmem_debug.o hakmem_sys.o hakmem_whale.o hakmem_policy.o hakmem_ace.o hakmem_ace_stats.o hakmem_prof.o hakmem_learner.o hakmem_size_hist.o hakmem_learn_log.o hakmem_syscall.o hakmem_ace_metrics.o hakmem_ace_ucb1.o hakmem_ace_controller.o tiny_fastcache.o core/box/superslab_expansion_box.o core/box/integrity_box.o core/box/free_local_box.o core/box/free_remote_box.o core/box/free_publish_box.o core/box/mailbox_box.o core/box/front_gate_box.o core/box/front_gate_classifier.o core/box/capacity_box.o core/box/carve_push_box.o core/box/prewarm_box.o core/box/ss_hot_prewarm_box.o core/box/front_metrics_box.o core/box/bench_fast_box.o core/link_stubs.o core/tiny_failfast.o bench_allocators_hakmem.o
|
||||
BENCH_HAKMEM_OBJS_BASE = hakmem.o hakmem_config.o hakmem_tiny_config.o hakmem_ucb1.o hakmem_bigcache.o hakmem_pool.o hakmem_l25_pool.o hakmem_site_rules.o hakmem_tiny.o hakmem_tiny_superslab.o hakmem_smallmid.o tiny_sticky.o tiny_remote.o tiny_publish.o tiny_debug_ring.o hakmem_tiny_magazine.o hakmem_tiny_stats.o hakmem_tiny_sfc.o hakmem_tiny_query.o hakmem_tiny_rss.o hakmem_tiny_registry.o hakmem_tiny_remote_target.o hakmem_tiny_bg_spill.o tiny_adaptive_sizing.o hakmem_mid_mt.o hakmem_super_registry.o hakmem_elo.o hakmem_batch.o hakmem_p2.o hakmem_sizeclass_dist.o hakmem_evo.o hakmem_debug.o hakmem_sys.o hakmem_whale.o hakmem_policy.o hakmem_ace.o hakmem_ace_stats.o hakmem_prof.o hakmem_learner.o hakmem_size_hist.o hakmem_learn_log.o hakmem_syscall.o hakmem_ace_metrics.o hakmem_ace_ucb1.o hakmem_ace_controller.o tiny_fastcache.o core/box/superslab_expansion_box.o core/box/integrity_box.o core/box/free_local_box.o core/box/free_remote_box.o core/box/free_publish_box.o core/box/mailbox_box.o core/box/front_gate_box.o core/box/front_gate_classifier.o core/box/capacity_box.o core/box/carve_push_box.o core/box/prewarm_box.o core/box/ss_hot_prewarm_box.o core/box/front_metrics_box.o core/box/bench_fast_box.o core/front/tiny_ring_cache.o core/link_stubs.o core/tiny_failfast.o bench_allocators_hakmem.o
|
||||
BENCH_HAKMEM_OBJS = $(BENCH_HAKMEM_OBJS_BASE)
|
||||
ifeq ($(POOL_TLS_PHASE1),1)
|
||||
BENCH_HAKMEM_OBJS += pool_tls.o pool_refill.o pool_tls_arena.o pool_tls_registry.o pool_tls_remote.o
|
||||
|
||||
161
core/front/tiny_ring_cache.c
Normal file
161
core/front/tiny_ring_cache.c
Normal file
@ -0,0 +1,161 @@
|
||||
// tiny_ring_cache.c - Phase 21-1: Ring cache implementation
|
||||
#include "tiny_ring_cache.h"
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
// ============================================================================
|
||||
// TLS Variables (defined here, extern in header)
|
||||
// ============================================================================
|
||||
|
||||
__thread TinyRingCache g_ring_cache_c2 = {NULL, 0, 0, 0, 0};
|
||||
__thread TinyRingCache g_ring_cache_c3 = {NULL, 0, 0, 0, 0};
|
||||
|
||||
// ============================================================================
|
||||
// Init (called at thread start, from hakmem_tiny.c)
|
||||
// ============================================================================
|
||||
|
||||
void ring_cache_init(void) {
|
||||
if (!ring_cache_enabled()) return;
|
||||
|
||||
// C2 init
|
||||
size_t cap_c2 = ring_capacity_c2();
|
||||
g_ring_cache_c2.slots = (void**)calloc(cap_c2, sizeof(void*));
|
||||
if (!g_ring_cache_c2.slots) {
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-INIT] Failed to allocate C2 ring (%zu slots)\n", cap_c2);
|
||||
fflush(stderr);
|
||||
#endif
|
||||
return;
|
||||
}
|
||||
g_ring_cache_c2.capacity = (uint16_t)cap_c2;
|
||||
g_ring_cache_c2.mask = (uint16_t)(cap_c2 - 1);
|
||||
g_ring_cache_c2.head = 0;
|
||||
g_ring_cache_c2.tail = 0;
|
||||
|
||||
// C3 init
|
||||
size_t cap_c3 = ring_capacity_c3();
|
||||
g_ring_cache_c3.slots = (void**)calloc(cap_c3, sizeof(void*));
|
||||
if (!g_ring_cache_c3.slots) {
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-INIT] Failed to allocate C3 ring (%zu slots)\n", cap_c3);
|
||||
fflush(stderr);
|
||||
#endif
|
||||
// Free C2 if C3 failed
|
||||
free(g_ring_cache_c2.slots);
|
||||
g_ring_cache_c2.slots = NULL;
|
||||
return;
|
||||
}
|
||||
g_ring_cache_c3.capacity = (uint16_t)cap_c3;
|
||||
g_ring_cache_c3.mask = (uint16_t)(cap_c3 - 1);
|
||||
g_ring_cache_c3.head = 0;
|
||||
g_ring_cache_c3.tail = 0;
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-INIT] C2=%zu slots (%zu bytes), C3=%zu slots (%zu bytes)\n",
|
||||
cap_c2, cap_c2 * sizeof(void*),
|
||||
cap_c3, cap_c3 * sizeof(void*));
|
||||
fflush(stderr);
|
||||
#endif
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Shutdown (called at thread exit, optional)
|
||||
// ============================================================================
|
||||
|
||||
void ring_cache_shutdown(void) {
|
||||
if (!ring_cache_enabled()) return;
|
||||
|
||||
// Drain rings to TLS SLL before shutdown (prevent leak)
|
||||
// TODO: Implement drain logic in Phase 21-1-C
|
||||
|
||||
// Free ring buffers
|
||||
if (g_ring_cache_c2.slots) {
|
||||
free(g_ring_cache_c2.slots);
|
||||
g_ring_cache_c2.slots = NULL;
|
||||
}
|
||||
|
||||
if (g_ring_cache_c3.slots) {
|
||||
free(g_ring_cache_c3.slots);
|
||||
g_ring_cache_c3.slots = NULL;
|
||||
}
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-SHUTDOWN] C2/C3 rings freed\n");
|
||||
fflush(stderr);
|
||||
#endif
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Refill from TLS SLL (cascade, Phase 21-1-C)
|
||||
// ============================================================================
|
||||
|
||||
// Refill ring from TLS SLL (one-way cascade: SLL → Ring)
|
||||
// Returns: number of blocks transferred
|
||||
int ring_refill_from_sll(int class_idx, int target_count) {
|
||||
if (!ring_cascade_enabled()) return 0;
|
||||
if (class_idx != 2 && class_idx != 3) return 0;
|
||||
|
||||
// Forward declarations (external functions from tls_sll_box.h)
|
||||
extern int tls_sll_pop(int class_idx, void** out_ptr);
|
||||
extern int tls_sll_push(int class_idx, void* ptr, uint32_t capacity);
|
||||
|
||||
int transferred = 0;
|
||||
|
||||
while (transferred < target_count) {
|
||||
void* ptr = NULL;
|
||||
|
||||
// Pop from TLS SLL
|
||||
if (!tls_sll_pop(class_idx, &ptr)) {
|
||||
break; // SLL empty
|
||||
}
|
||||
|
||||
// Push to Ring
|
||||
if (!ring_cache_push(class_idx, ptr)) {
|
||||
// Ring full, push back to SLL
|
||||
tls_sll_push(class_idx, ptr, (uint32_t)-1); // Unlimited capacity
|
||||
break;
|
||||
}
|
||||
|
||||
transferred++;
|
||||
}
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
if (transferred > 0) {
|
||||
fprintf(stderr, "[Ring-REFILL] C%d: %d blocks transferred from SLL to Ring\n",
|
||||
class_idx, transferred);
|
||||
fflush(stderr);
|
||||
}
|
||||
#endif
|
||||
|
||||
return transferred;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Stats (Phase 19-1 metrics, TODO)
|
||||
// ============================================================================
|
||||
|
||||
void ring_cache_print_stats(void) {
|
||||
if (!ring_cache_enabled()) return;
|
||||
|
||||
// TODO: Add metrics tracking (Phase 21-1-E)
|
||||
// - Hit rate (Ring hits / total allocs)
|
||||
// - Miss rate (Ring empty / total allocs)
|
||||
// - Full rate (Ring full / total frees)
|
||||
// - Refill count (SLL → Ring transfers)
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
// Current occupancy
|
||||
uint16_t c2_count = (g_ring_cache_c2.tail >= g_ring_cache_c2.head)
|
||||
? (g_ring_cache_c2.tail - g_ring_cache_c2.head)
|
||||
: (g_ring_cache_c2.capacity - g_ring_cache_c2.head + g_ring_cache_c2.tail);
|
||||
|
||||
uint16_t c3_count = (g_ring_cache_c3.tail >= g_ring_cache_c3.head)
|
||||
? (g_ring_cache_c3.tail - g_ring_cache_c3.head)
|
||||
: (g_ring_cache_c3.capacity - g_ring_cache_c3.head + g_ring_cache_c3.tail);
|
||||
|
||||
fprintf(stderr, "[Ring-STATS] C2: %u/%u slots, C3: %u/%u slots\n",
|
||||
c2_count, g_ring_cache_c2.capacity,
|
||||
c3_count, g_ring_cache_c3.capacity);
|
||||
fflush(stderr);
|
||||
#endif
|
||||
}
|
||||
189
core/front/tiny_ring_cache.h
Normal file
189
core/front/tiny_ring_cache.h
Normal file
@ -0,0 +1,189 @@
|
||||
// tiny_ring_cache.h - Phase 21-1: Array-based hot cache (C2/C3 only)
|
||||
//
|
||||
// Goal: Eliminate pointer chasing in TLS SLL by using ring buffer
|
||||
// Target: +15-20% performance (54.4M → 62-65M ops/s)
|
||||
//
|
||||
// Design (ChatGPT feedback):
|
||||
// - Ring → SLL → SuperSlab (3-layer hierarchy)
|
||||
// - Ring size: 128 slots (ENV: 64/128/256 A/B test)
|
||||
// - C2/C3 only (hot classes, 33-128B)
|
||||
// - Replaces UltraHot (Phase 19-3: +12.9% by removing UltraHot)
|
||||
//
|
||||
// Performance:
|
||||
// - Alloc: 1-2 instructions (array access, no pointer chasing)
|
||||
// - Free: 1-2 instructions (array write, no pointer chasing)
|
||||
// - vs TLS SLL: 3 mem accesses → 2 mem accesses, 1 cache miss → 0
|
||||
//
|
||||
// ENV Variables:
|
||||
// HAKMEM_TINY_HOT_RING_ENABLE=1 # Enable Ring cache (default: 0)
|
||||
// HAKMEM_TINY_HOT_RING_C2=128 # C2 ring size (default: 128)
|
||||
// HAKMEM_TINY_HOT_RING_C3=128 # C3 ring size (default: 128)
|
||||
// HAKMEM_TINY_HOT_RING_CASCADE=1 # Enable SLL → Ring refill (default: 0)
|
||||
|
||||
#ifndef HAK_FRONT_TINY_RING_CACHE_H
|
||||
#define HAK_FRONT_TINY_RING_CACHE_H
|
||||
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include "../hakmem_build_flags.h"
|
||||
|
||||
// ============================================================================
|
||||
// Ring Buffer Structure
|
||||
// ============================================================================
|
||||
|
||||
typedef struct {
|
||||
void** slots; // Dynamic array (allocated at init, power-of-2 size)
|
||||
uint16_t head; // Pop index (consumer)
|
||||
uint16_t tail; // Push index (producer)
|
||||
uint16_t capacity; // Ring size (power of 2 for fast modulo: & (capacity-1))
|
||||
uint16_t mask; // Capacity - 1 (for fast modulo)
|
||||
} TinyRingCache;
|
||||
|
||||
// ============================================================================
|
||||
// External TLS Variables (defined in hakmem_tiny.c)
|
||||
// ============================================================================
|
||||
|
||||
extern __thread TinyRingCache g_ring_cache_c2;
|
||||
extern __thread TinyRingCache g_ring_cache_c3;
|
||||
|
||||
// ============================================================================
|
||||
// ENV Control (cached, lazy init)
|
||||
// ============================================================================
|
||||
|
||||
// Enable flag (default: 0, OFF)
|
||||
static inline int ring_cache_enabled(void) {
|
||||
static int g_enable = -1;
|
||||
if (__builtin_expect(g_enable == -1, 0)) {
|
||||
const char* e = getenv("HAKMEM_TINY_HOT_RING_ENABLE");
|
||||
g_enable = (e && *e && *e != '0') ? 1 : 0;
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
if (g_enable) {
|
||||
fprintf(stderr, "[Ring-INIT] ring_cache_enabled() = %d\n", g_enable);
|
||||
fflush(stderr);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
return g_enable;
|
||||
}
|
||||
|
||||
// C2 capacity (default: 128)
|
||||
static inline size_t ring_capacity_c2(void) {
|
||||
static size_t g_cap = 0;
|
||||
if (__builtin_expect(g_cap == 0, 0)) {
|
||||
const char* e = getenv("HAKMEM_TINY_HOT_RING_C2");
|
||||
g_cap = (e && *e) ? (size_t)atoi(e) : 128; // Default: 128
|
||||
|
||||
// Round up to power of 2 (for fast modulo)
|
||||
if (g_cap < 32) g_cap = 32;
|
||||
if (g_cap > 256) g_cap = 256;
|
||||
|
||||
// Ensure power of 2
|
||||
size_t pow2 = 32;
|
||||
while (pow2 < g_cap) pow2 *= 2;
|
||||
g_cap = pow2;
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-INIT] C2 capacity = %zu (power of 2)\n", g_cap);
|
||||
fflush(stderr);
|
||||
#endif
|
||||
}
|
||||
return g_cap;
|
||||
}
|
||||
|
||||
// C3 capacity (default: 128)
|
||||
static inline size_t ring_capacity_c3(void) {
|
||||
static size_t g_cap = 0;
|
||||
if (__builtin_expect(g_cap == 0, 0)) {
|
||||
const char* e = getenv("HAKMEM_TINY_HOT_RING_C3");
|
||||
g_cap = (e && *e) ? (size_t)atoi(e) : 128; // Default: 128
|
||||
|
||||
// Round up to power of 2
|
||||
if (g_cap < 32) g_cap = 32;
|
||||
if (g_cap > 256) g_cap = 256;
|
||||
|
||||
size_t pow2 = 32;
|
||||
while (pow2 < g_cap) pow2 *= 2;
|
||||
g_cap = pow2;
|
||||
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
fprintf(stderr, "[Ring-INIT] C3 capacity = %zu (power of 2)\n", g_cap);
|
||||
fflush(stderr);
|
||||
#endif
|
||||
}
|
||||
return g_cap;
|
||||
}
|
||||
|
||||
// Cascade enable flag (default: 0, OFF)
|
||||
static inline int ring_cascade_enabled(void) {
|
||||
static int g_enable = -1;
|
||||
if (__builtin_expect(g_enable == -1, 0)) {
|
||||
const char* e = getenv("HAKMEM_TINY_HOT_RING_CASCADE");
|
||||
g_enable = (e && *e && *e != '0') ? 1 : 0;
|
||||
#if !HAKMEM_BUILD_RELEASE
|
||||
if (g_enable) {
|
||||
fprintf(stderr, "[Ring-INIT] ring_cascade_enabled() = %d\n", g_enable);
|
||||
fflush(stderr);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
return g_enable;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Ultra-Fast Pop/Push (1-2 instructions)
|
||||
// ============================================================================
|
||||
|
||||
// Pop from ring (alloc fast path)
|
||||
// Returns: BASE pointer (caller must convert to USER with +1)
|
||||
static inline void* ring_cache_pop(int class_idx) {
|
||||
TinyRingCache* ring = (class_idx == 2) ? &g_ring_cache_c2 : &g_ring_cache_c3;
|
||||
|
||||
// Empty check
|
||||
if (__builtin_expect(ring->head == ring->tail, 0)) {
|
||||
return NULL; // Empty
|
||||
}
|
||||
|
||||
// Pop from head (consumer)
|
||||
void* base = ring->slots[ring->head];
|
||||
ring->head = (ring->head + 1) & ring->mask; // Fast modulo (power of 2)
|
||||
|
||||
return base; // Return BASE pointer
|
||||
}
|
||||
|
||||
// Push to ring (free fast path)
|
||||
// Input: BASE pointer (caller must pass BASE, not USER)
|
||||
// Returns: 1=SUCCESS, 0=FULL
|
||||
static inline int ring_cache_push(int class_idx, void* base) {
|
||||
TinyRingCache* ring = (class_idx == 2) ? &g_ring_cache_c2 : &g_ring_cache_c3;
|
||||
|
||||
uint16_t next_tail = (ring->tail + 1) & ring->mask;
|
||||
|
||||
// Full check (leave 1 slot empty to distinguish full/empty)
|
||||
if (__builtin_expect(next_tail == ring->head, 0)) {
|
||||
return 0; // Full
|
||||
}
|
||||
|
||||
// Push to tail (producer)
|
||||
ring->slots[ring->tail] = base;
|
||||
ring->tail = next_tail;
|
||||
|
||||
return 1; // SUCCESS
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Refill from TLS SLL (cascade, Phase 21-1-C)
|
||||
// ============================================================================
|
||||
|
||||
// Forward declaration (defined in tiny_ring_cache.c)
|
||||
int ring_refill_from_sll(int class_idx, int target_count);
|
||||
|
||||
// ============================================================================
|
||||
// Init/Shutdown (called from hakmem_tiny.c)
|
||||
// ============================================================================
|
||||
|
||||
void ring_cache_init(void);
|
||||
void ring_cache_shutdown(void);
|
||||
void ring_cache_print_stats(void);
|
||||
|
||||
#endif // HAK_FRONT_TINY_RING_CACHE_H
|
||||
Reference in New Issue
Block a user