Phase 75-6: define SSOT policy to avoid baseline drift

This commit is contained in:
Moe Charm (CI)
2025-12-18 10:22:24 +09:00
parent 9123a8f12b
commit d5c1113b4c
2 changed files with 67 additions and 0 deletions

View File

@ -41,6 +41,7 @@
- 変更は必ず **箱 + 境界1箇所 + ENVで戻せる** で積むFail-fast、最小可視化 - 変更は必ず **箱 + 境界1箇所 + ENVで戻せる** で積むFail-fast、最小可視化
- A/B は **同一バイナリでENVトグル**が原則(別バイナリ比較は layout が混ざる)。 - A/B は **同一バイナリでENVトグル**が原則(別バイナリ比較は layout が混ざる)。
- SSOT運用ころころ防止: `docs/analysis/PHASE75_6_SSOT_POLICY_FAST_PGO_VS_STANDARD.md`
- “削除して速い” は封印link-out/大削除は layout tax で符号反転しやすい)→ **compile-out** を優先。 - “削除して速い” は封印link-out/大削除は layout tax で符号反転しやすい)→ **compile-out** を優先。
- 診断: `scripts/box/layout_tax_forensics_box.sh` / `docs/analysis/PHASE67A_LAYOUT_TAX_FORENSICS_SSOT.md` - 診断: `scripts/box/layout_tax_forensics_box.sh` / `docs/analysis/PHASE67A_LAYOUT_TAX_FORENSICS_SSOT.md`

View File

@ -0,0 +1,66 @@
# Phase 75-6: SSOT Policy — FAST PGO vs Standard (stop “ころころ” drift)
## Problem statement
After Phase 75, we observed:
- Phase 75 win is **real** (C5/C6 inline slots improve D vs A in both Standard and FAST PGO).
- Absolute “baseline” numbers **move** across commits/builds (especially with PGO), causing SSOT confusion (“ころころ変わる”).
This document defines a stable SSOT policy that keeps Box Theory iteration reliable.
## Definitions
### Standard binary
- `./bench_random_mixed_hakmem`
- Used for: correctness, production-like behavior, “stable across code refactors”
### FAST PGO binary
- `./bench_random_mixed_hakmem_minimal_pgo`
- Used for: competitive speed tracking vs mimalloc (best-case tuned build)
- Caveat: more sensitive to build/layout drift than Standard
### SSOT harness
- `scripts/run_mixed_10_cleanenv.sh`
- Must pin the binary explicitly via `BENCH_BIN=...` when comparing Standard vs FAST.
## SSOT policy (two-track)
### Track A (Decision SSOT): same-binary A/B
For accepting a feature (GO/NEUTRAL/NO-GO), the primary truth is:
- **same binary**, **ENV toggle only**
- Example: Phase 75 4-point matrix within the same binary.
This avoids layout tax from “different binaries” and is aligned with prior learnings:
- link-out / large pruning can flip signs due to layout.
### Track B (Competitive SSOT): FAST PGO ratio vs mimalloc
For “how close to mimalloc”, use FAST PGO:
- `BENCH_BIN=./bench_random_mixed_hakmem_minimal_pgo`
- mimalloc is still a separate binary reference (layout differs), so treat ratio as “headline”, not proof of a micro-change.
## Practical rules to prevent SSOT drift
1. **Never mix Standard numbers into FAST ratio tables**
- Standard A/B results are valid, but not directly comparable to FAST baseline.
2. **When reporting a result, always include:**
- binary (`bench_random_mixed_hakmem` vs `bench_random_mixed_hakmem_minimal_pgo`)
- workload (`ITERS`, `WS`, `RUNS`)
- key ENV knobs (`WARM_POOL_SIZE`, `C5/C6 inline`, etc.)
3. **If FAST PGO baseline changes across commits**
- treat it as “baseline rebase event”, not automatically “regression”
- confirm using `scripts/box/layout_tax_forensics_box.sh` + perf stat deltas (IPC/branch/cache)
4. **Do not demote FAST PGO SSOT solely from one episode**
- use Track A (same-binary A/B) to validate the optimization first
- then decide whether FAST PGO is “worth maintaining” based on ongoing ROI
## Recommended next action after Phase 75-5
- Keep Phase 75 (C5/C6) promoted for Standard and for FAST builds.
- Treat Phase 69s 62.63M as historical reference, not guaranteed to reproduce on later commits.
- Proceed with Phase 76 using Track A for GO decisions, and Track B for periodic headline updates.