Phase 75-6: define SSOT policy to avoid baseline drift
This commit is contained in:
@ -41,6 +41,7 @@
|
||||
|
||||
- 変更は必ず **箱 + 境界1箇所 + ENVで戻せる** で積む(Fail-fast、最小可視化)。
|
||||
- A/B は **同一バイナリでENVトグル**が原則(別バイナリ比較は layout が混ざる)。
|
||||
- SSOT運用(ころころ防止): `docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md`
|
||||
- “削除して速い” は封印(link-out/大削除は layout tax で符号反転しやすい)→ **compile-out** を優先。
|
||||
- 診断: `scripts/box/layout_tax_forensics_box.sh` / `docs/analysis/PHASE67A_LAYOUT_TAX_FORENSICS_SSOT.md`
|
||||
|
||||
|
||||
66
docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md
Normal file
66
docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md
Normal file
@ -0,0 +1,66 @@
|
||||
# Phase 75-6: SSOT Policy — FAST PGO vs Standard (stop “ころころ” drift)
|
||||
|
||||
## Problem statement
|
||||
|
||||
After Phase 75, we observed:
|
||||
- Phase 75 win is **real** (C5/C6 inline slots improve D vs A in both Standard and FAST PGO).
|
||||
- Absolute “baseline” numbers **move** across commits/builds (especially with PGO), causing SSOT confusion (“ころころ変わる”).
|
||||
|
||||
This document defines a stable SSOT policy that keeps Box Theory iteration reliable.
|
||||
|
||||
## Definitions
|
||||
|
||||
### Standard binary
|
||||
- `./bench_random_mixed_hakmem`
|
||||
- Used for: correctness, production-like behavior, “stable across code refactors”
|
||||
|
||||
### FAST PGO binary
|
||||
- `./bench_random_mixed_hakmem_minimal_pgo`
|
||||
- Used for: competitive speed tracking vs mimalloc (best-case tuned build)
|
||||
- Caveat: more sensitive to build/layout drift than Standard
|
||||
|
||||
### SSOT harness
|
||||
- `scripts/run_mixed_10_cleanenv.sh`
|
||||
- Must pin the binary explicitly via `BENCH_BIN=...` when comparing Standard vs FAST.
|
||||
|
||||
## SSOT policy (two-track)
|
||||
|
||||
### Track A (Decision SSOT): same-binary A/B
|
||||
|
||||
For accepting a feature (GO/NEUTRAL/NO-GO), the primary truth is:
|
||||
- **same binary**, **ENV toggle only**
|
||||
- Example: Phase 75 4-point matrix within the same binary.
|
||||
|
||||
This avoids layout tax from “different binaries” and is aligned with prior learnings:
|
||||
- link-out / large pruning can flip signs due to layout.
|
||||
|
||||
### Track B (Competitive SSOT): FAST PGO ratio vs mimalloc
|
||||
|
||||
For “how close to mimalloc”, use FAST PGO:
|
||||
- `BENCH_BIN=./bench_random_mixed_hakmem_minimal_pgo`
|
||||
- mimalloc is still a separate binary reference (layout differs), so treat ratio as “headline”, not proof of a micro-change.
|
||||
|
||||
## Practical rules to prevent SSOT drift
|
||||
|
||||
1. **Never mix Standard numbers into FAST ratio tables**
|
||||
- Standard A/B results are valid, but not directly comparable to FAST baseline.
|
||||
|
||||
2. **When reporting a result, always include:**
|
||||
- binary (`bench_random_mixed_hakmem` vs `bench_random_mixed_hakmem_minimal_pgo`)
|
||||
- workload (`ITERS`, `WS`, `RUNS`)
|
||||
- key ENV knobs (`WARM_POOL_SIZE`, `C5/C6 inline`, etc.)
|
||||
|
||||
3. **If FAST PGO baseline changes across commits**
|
||||
- treat it as “baseline rebase event”, not automatically “regression”
|
||||
- confirm using `scripts/box/layout_tax_forensics_box.sh` + perf stat deltas (IPC/branch/cache)
|
||||
|
||||
4. **Do not demote FAST PGO SSOT solely from one episode**
|
||||
- use Track A (same-binary A/B) to validate the optimization first
|
||||
- then decide whether FAST PGO is “worth maintaining” based on ongoing ROI
|
||||
|
||||
## Recommended next action after Phase 75-5
|
||||
|
||||
- Keep Phase 75 (C5/C6) promoted for Standard and for FAST builds.
|
||||
- Treat Phase 69’s 62.63M as historical reference, not guaranteed to reproduce on later commits.
|
||||
- Proceed with Phase 76 using Track A for GO decisions, and Track B for periodic headline updates.
|
||||
|
||||
Reference in New Issue
Block a user