tomoaki
1f3ef2ac61
Phase 287 P2 completed: 45-second target achieved Changes: - Moved 34 heavy tests (>0.4s) from quick to integration - Disabled FAST_FAIL in quick profile for accurate measurement - Reduced execution time: 63.0s → 45.85s (27% reduction) - Reduced test count: 447 → 413 tests (8% reduction) Key improvements: - Fixed FAST_FAIL=1 issue preventing full test execution - Selective individual test movement vs directory-level in P1 - Maintained relative path structure for --filter compatibility Performance metrics: - P1 (FAST_FAIL=0): 63.0s / 447 tests / 111 failures - P2 (optimized): 45.85s / 413 tests / 90 failures - Target achieved: ✅ 45s (within tolerance) Moved test categories: - mirbuilder_loop_* (0.46-0.49s) - mirbuilder_provider_* (0.36-0.68s) - hako_primary_no_fallback_* (0.46-0.47s) - parser_embedded_json_canary (0.49s) - emit_mir_canary (0.47s) Documentation updates: - tools/smokes/v2/README.md: Updated quick target to ~400 tests - docs/development/current/main/phases/phase-287/README.md: P2 results - tools/smokes/v2/configs/auto_detect.conf: FAST_FAIL=0 for accurate timing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
54 lines
3.1 KiB
Markdown
54 lines
3.1 KiB
Markdown
Smokes v2 — Minimal Runner and Policy
|
||
|
||
## Profiles
|
||
|
||
### quick
|
||
- **Purpose**: Fast feedback for development
|
||
- **Target**: ~45 seconds, ~400 tests (Phase 287 P2: 45.85s / 413 tests ✅)
|
||
- **Contents**: Minimal gate tests - basic syntax, using resolution, essential control flow
|
||
- **Excluded**: Heavy selfhost/Stage-B, crate exe, S3/LLVM integration, long-running tests (>0.4s)
|
||
|
||
### integration
|
||
- **Purpose**: Integration and heavier tests
|
||
- **Contents**: Selfhost canaries, Stage-B, crate exe, S3/LLVM, phase-specific comprehensive tests
|
||
- **Run**: `./tools/smokes/v2/run.sh --profile integration`
|
||
|
||
### full
|
||
- **Purpose**: Complete regression coverage
|
||
- **Contents**: All tests (network of quick + integration + additional)
|
||
- **Run**: `./tools/smokes/v2/run.sh --profile full`
|
||
|
||
### plugins
|
||
- **Purpose**: Plugin-specific tests (dynamic loading, etc.)
|
||
- **Run**: `./tools/smokes/v2/run.sh --profile plugins`
|
||
|
||
Policy
|
||
- Use [SKIP:<reason>] prefix for environment/host dependent skips.
|
||
- Examples: [SKIP] hakorune not built, [SKIP:env] plugin path missing
|
||
- Keep reasons short and stable to allow grep-based canaries.
|
||
- Prefer JSON-only output in CI: set `NYASH_JSON_ONLY=1` to avoid noisy logs.
|
||
- Diagnostics lines like `[provider/select:*]` are filtered by default in `lib/test_runner.sh`.
|
||
- Toggle: set `HAKO_SILENT_TAGS=0` to disable filtering and show raw logs. `HAKO_SHOW_CALL_LOGS=1` also bypasses filtering.
|
||
|
||
Helpers
|
||
- `tools/smokes/v2/lib/mir_canary.sh` provides:
|
||
- `extract_mir_from_output` — between [MIR_BEGIN]/[MIR_END]
|
||
- `assert_has_tokens`, `assert_skip_tag`, `assert_order`, `assert_token_count`
|
||
- `tools/lib/canary.sh` provides minimal, harness-agnostic aliases:
|
||
- `extract_mir_between_tags` — same as `extract_mir_from_output`
|
||
- `require_tokens token...` — fail if any token missing
|
||
|
||
Notes
|
||
- Avoid running heavy integration smokes in CI by default. Use `--profile quick`.
|
||
- When a test depends on external tools (e.g., LLVM), prefer `[SKIP:<reason>]` over failure.
|
||
- Stage‑B/selfhost canaries（`stage1_launcher_*`, `phase251*` など）は Stage‑3 デフォルト環境で安定しないため、quick プロファイルでは `[SKIP:stageb]` として扱い、必要に応じて別プロファイル（integration/full）で個別に実行する。
|
||
- Selfhost quick カバレッジは最小 1 本（`core/selfhost_minimal.sh`）に絞り、Stage‑3 + JoinIR 前提で Stage‑B→VM を通るかだけを確認する。
|
||
- S3 backend 向けの長尺テスト群も quick 向きではないため、timeout を短く保ちたい場合は `[SKIP:slow]` にして別途ローカルで回すことを推奨する。
|
||
|
||
Quick tips
|
||
- EXE-heavy cases (e.g., `phase2100/*`) may take longer. When running quick with these tests, pass a larger timeout like `--timeout 120`.
|
||
- Smokes v2 auto-cleans temporary crate EXE objects created under `/tmp` (pattern: `ny_crate_backend_exe_*.o`) after the run.
|
||
|
||
Developer Notes
|
||
- **JoinIR If/Select (Phase 33)**: A/B test with `NYASH_FEATURES=stage3 HAKO_JOINIR_IF_SELECT=1 ./target/release/hakorune apps/tests/joinir_if_select_simple.hako`（dev-only、CI対象外。NYASH_JOINIR_CORE は deprecated/無視）
|