Phase 75-6: define SSOT policy to avoid baseline drift
This commit is contained in:
@ -41,6 +41,7 @@
|
|||||||
|
|
||||||
- 変更は必ず **箱 + 境界1箇所 + ENVで戻せる** で積む(Fail-fast、最小可視化)。
|
- 変更は必ず **箱 + 境界1箇所 + ENVで戻せる** で積む(Fail-fast、最小可視化)。
|
||||||
- A/B は **同一バイナリでENVトグル**が原則(別バイナリ比較は layout が混ざる)。
|
- A/B は **同一バイナリでENVトグル**が原則(別バイナリ比較は layout が混ざる)。
|
||||||
|
- SSOT運用(ころころ防止): `docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md`
|
||||||
- “削除して速い” は封印(link-out/大削除は layout tax で符号反転しやすい)→ **compile-out** を優先。
|
- “削除して速い” は封印(link-out/大削除は layout tax で符号反転しやすい)→ **compile-out** を優先。
|
||||||
- 診断: `scripts/box/layout_tax_forensics_box.sh` / `docs/analysis/PHASE67A_LAYOUT_TAX_FORENSICS_SSOT.md`
|
- 診断: `scripts/box/layout_tax_forensics_box.sh` / `docs/analysis/PHASE67A_LAYOUT_TAX_FORENSICS_SSOT.md`
|
||||||
|
|
||||||
|
|||||||
66
docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md
Normal file
66
docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md
Normal file
@ -0,0 +1,66 @@
|
|||||||
|
# Phase 75-6: SSOT Policy — FAST PGO vs Standard (stop “ころころ” drift)
|
||||||
|
|
||||||
|
## Problem statement
|
||||||
|
|
||||||
|
After Phase 75, we observed:
|
||||||
|
- Phase 75 win is **real** (C5/C6 inline slots improve D vs A in both Standard and FAST PGO).
|
||||||
|
- Absolute “baseline” numbers **move** across commits/builds (especially with PGO), causing SSOT confusion (“ころころ変わる”).
|
||||||
|
|
||||||
|
This document defines a stable SSOT policy that keeps Box Theory iteration reliable.
|
||||||
|
|
||||||
|
## Definitions
|
||||||
|
|
||||||
|
### Standard binary
|
||||||
|
- `./bench_random_mixed_hakmem`
|
||||||
|
- Used for: correctness, production-like behavior, “stable across code refactors”
|
||||||
|
|
||||||
|
### FAST PGO binary
|
||||||
|
- `./bench_random_mixed_hakmem_minimal_pgo`
|
||||||
|
- Used for: competitive speed tracking vs mimalloc (best-case tuned build)
|
||||||
|
- Caveat: more sensitive to build/layout drift than Standard
|
||||||
|
|
||||||
|
### SSOT harness
|
||||||
|
- `scripts/run_mixed_10_cleanenv.sh`
|
||||||
|
- Must pin the binary explicitly via `BENCH_BIN=...` when comparing Standard vs FAST.
|
||||||
|
|
||||||
|
## SSOT policy (two-track)
|
||||||
|
|
||||||
|
### Track A (Decision SSOT): same-binary A/B
|
||||||
|
|
||||||
|
For accepting a feature (GO/NEUTRAL/NO-GO), the primary truth is:
|
||||||
|
- **same binary**, **ENV toggle only**
|
||||||
|
- Example: Phase 75 4-point matrix within the same binary.
|
||||||
|
|
||||||
|
This avoids layout tax from “different binaries” and is aligned with prior learnings:
|
||||||
|
- link-out / large pruning can flip signs due to layout.
|
||||||
|
|
||||||
|
### Track B (Competitive SSOT): FAST PGO ratio vs mimalloc
|
||||||
|
|
||||||
|
For “how close to mimalloc”, use FAST PGO:
|
||||||
|
- `BENCH_BIN=./bench_random_mixed_hakmem_minimal_pgo`
|
||||||
|
- mimalloc is still a separate binary reference (layout differs), so treat ratio as “headline”, not proof of a micro-change.
|
||||||
|
|
||||||
|
## Practical rules to prevent SSOT drift
|
||||||
|
|
||||||
|
1. **Never mix Standard numbers into FAST ratio tables**
|
||||||
|
- Standard A/B results are valid, but not directly comparable to FAST baseline.
|
||||||
|
|
||||||
|
2. **When reporting a result, always include:**
|
||||||
|
- binary (`bench_random_mixed_hakmem` vs `bench_random_mixed_hakmem_minimal_pgo`)
|
||||||
|
- workload (`ITERS`, `WS`, `RUNS`)
|
||||||
|
- key ENV knobs (`WARM_POOL_SIZE`, `C5/C6 inline`, etc.)
|
||||||
|
|
||||||
|
3. **If FAST PGO baseline changes across commits**
|
||||||
|
- treat it as “baseline rebase event”, not automatically “regression”
|
||||||
|
- confirm using `scripts/box/layout_tax_forensics_box.sh` + perf stat deltas (IPC/branch/cache)
|
||||||
|
|
||||||
|
4. **Do not demote FAST PGO SSOT solely from one episode**
|
||||||
|
- use Track A (same-binary A/B) to validate the optimization first
|
||||||
|
- then decide whether FAST PGO is “worth maintaining” based on ongoing ROI
|
||||||
|
|
||||||
|
## Recommended next action after Phase 75-5
|
||||||
|
|
||||||
|
- Keep Phase 75 (C5/C6) promoted for Standard and for FAST builds.
|
||||||
|
- Treat Phase 69’s 62.63M as historical reference, not guaranteed to reproduce on later commits.
|
||||||
|
- Proceed with Phase 76 using Track A for GO decisions, and Track B for periodic headline updates.
|
||||||
|
|
||||||
Reference in New Issue
Block a user