Front-Direct implementation: SS→FC direct refill + SLL complete bypass
## Summary
Implemented Front-Direct architecture with complete SLL bypass:
- Direct SuperSlab → FastCache refill (1-hop, bypasses SLL)
- SLL-free allocation/free paths when Front-Direct enabled
- Legacy path sealing (SLL inline opt-in, SFC cascade ENV-only)
## New Modules
- core/refill/ss_refill_fc.h (236 lines): Standard SS→FC refill entry point
- Remote drain → Freelist → Carve priority
- Header restoration for C1-C6 (NOT C0/C7)
- ENV: HAKMEM_TINY_P0_DRAIN_THRESH, HAKMEM_TINY_P0_NO_DRAIN
- core/front/fast_cache.h: FastCache (L1) type definition
- core/front/quick_slot.h: QuickSlot (L0) type definition
## Allocation Path (core/tiny_alloc_fast.inc.h)
- Added s_front_direct_alloc TLS flag (lazy ENV check)
- SLL pop guarded by: g_tls_sll_enable && !s_front_direct_alloc
- Refill dispatch:
- Front-Direct: ss_refill_fc_fill() → fastcache_pop() (1-hop)
- Legacy: sll_refill_batch_from_ss() → SLL → FC (2-hop, A/B only)
- SLL inline pop sealed (requires HAKMEM_TINY_INLINE_SLL=1 opt-in)
## Free Path (core/hakmem_tiny_free.inc, core/hakmem_tiny_fastcache.inc.h)
- FC priority: Try fastcache_push() first (same-thread free)
- tiny_fast_push() bypass: Returns 0 when s_front_direct_free || !g_tls_sll_enable
- Fallback: Magazine/slow path (safe, bypasses SLL)
## Legacy Sealing
- SFC cascade: Default OFF (ENV-only via HAKMEM_TINY_SFC_CASCADE=1)
- Deleted: core/hakmem_tiny_free.inc.bak, core/pool_refill_legacy.c.bak
- Documentation: ss_refill_fc_fill() promoted as CANONICAL refill entry
## ENV Controls
- HAKMEM_TINY_FRONT_DIRECT=1: Enable Front-Direct (SS→FC direct)
- HAKMEM_TINY_P0_DIRECT_FC_ALL=1: Same as above (alt name)
- HAKMEM_TINY_REFILL_BATCH=1: Enable batch refill (also enables Front-Direct)
- HAKMEM_TINY_SFC_CASCADE=1: Enable SFC cascade (default OFF)
- HAKMEM_TINY_INLINE_SLL=1: Enable inline SLL pop (default OFF, requires AGGRESSIVE_INLINE)
## Benchmarks (Front-Direct Enabled)
```bash
ENV: HAKMEM_BENCH_FAST_FRONT=1 HAKMEM_TINY_FRONT_DIRECT=1
HAKMEM_TINY_REFILL_BATCH=1 HAKMEM_TINY_P0_DIRECT_FC_ALL=1
HAKMEM_TINY_REFILL_COUNT_HOT=256 HAKMEM_TINY_REFILL_COUNT_MID=96
HAKMEM_TINY_BUMP_CHUNK=256
bench_random_mixed (16-1040B random, 200K iter):
256 slots: 1.44M ops/s (STABLE, 0 SEGV)
128 slots: 1.44M ops/s (STABLE, 0 SEGV)
bench_fixed_size (fixed size, 200K iter):
256B: 4.06M ops/s (has debug logs, expected >10M without logs)
128B: Similar (debug logs affect)
```
## Verification
- TRACE_RING test (10K iter): **0 SLL events** detected ✅
- Complete SLL bypass confirmed when Front-Direct=1
- Stable execution: 200K iterations × multiple sizes, 0 SEGV
## Next Steps
- Disable debug logs in hak_alloc_api.inc.h (call_num 14250-14280 range)
- Re-benchmark with clean Release build (target: 10-15M ops/s)
- 128/256B shortcut path optimization (FC hit rate improvement)
Co-Authored-By: ChatGPT <chatgpt@openai.com>
Suggested-By: ultrathink
This commit is contained in:
@ -3,6 +3,7 @@
|
||||
#include <stdlib.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <unistd.h>
|
||||
#include "box/tiny_next_ptr_box.h" // Box API: preserve header by using class-aware next offset
|
||||
|
||||
#define REMOTE_BUCKETS 256
|
||||
|
||||
@ -34,7 +35,8 @@ int pool_remote_push(int class_idx, void* ptr, int owner_tid){
|
||||
r = (RemoteRec*)calloc(1, sizeof(RemoteRec));
|
||||
r->tid = owner_tid; r->next = g_buckets[b]; g_buckets[b] = r;
|
||||
}
|
||||
*(void**)ptr = r->head[class_idx];
|
||||
// Use Box next-pointer API to avoid clobbering header (classes 1-6 store next at base+1)
|
||||
tiny_next_write(class_idx, ptr, r->head[class_idx]);
|
||||
r->head[class_idx] = ptr;
|
||||
r->count[class_idx]++;
|
||||
pthread_mutex_unlock(&g_locks[b]);
|
||||
@ -57,9 +59,9 @@ int pool_remote_pop_chain(int class_idx, int max_take, void** out_chain){
|
||||
int batch = 0; if (max_take <= 0) max_take = 32;
|
||||
void* chain = NULL; void* tail = NULL;
|
||||
while (head && batch < max_take){
|
||||
void* nxt = *(void**)head;
|
||||
void* nxt = tiny_next_read(class_idx, head);
|
||||
if (!chain){ chain = head; tail = head; }
|
||||
else { *(void**)tail = head; tail = head; }
|
||||
else { tiny_next_write(class_idx, tail, head); tail = head; }
|
||||
head = nxt; batch++;
|
||||
}
|
||||
r->head[class_idx] = head;
|
||||
|
||||
Reference in New Issue
Block a user