tomoaki/hakorune
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
e360ea7de7461b57cf27734f474c7e02ba3a6761
hakorune/tools/smokes/v2/README.md
tomoaki 1f3ef2ac61 smokes(v2): Reach 45s target (Phase 287 P2) 
Phase 287 P2 completed: 45-second target achieved

Changes:
- Moved 34 heavy tests (>0.4s) from quick to integration
- Disabled FAST_FAIL in quick profile for accurate measurement
- Reduced execution time: 63.0s → 45.85s (27% reduction)
- Reduced test count: 447 → 413 tests (8% reduction)

Key improvements:
- Fixed FAST_FAIL=1 issue preventing full test execution
- Selective individual test movement vs directory-level in P1
- Maintained relative path structure for --filter compatibility

Performance metrics:
- P1 (FAST_FAIL=0): 63.0s / 447 tests / 111 failures
- P2 (optimized): 45.85s / 413 tests / 90 failures
- Target achieved: ✅ 45s (within tolerance)

Moved test categories:
- mirbuilder_loop_* (0.46-0.49s)
- mirbuilder_provider_* (0.36-0.68s)
- hako_primary_no_fallback_* (0.46-0.47s)
- parser_embedded_json_canary (0.49s)
- emit_mir_canary (0.47s)

Documentation updates:
- tools/smokes/v2/README.md: Updated quick target to ~400 tests
- docs/development/current/main/phases/phase-287/README.md: P2 results
- tools/smokes/v2/configs/auto_detect.conf: FAST_FAIL=0 for accurate timing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-25 07:58:51 +09:00

3.1 KiB
Raw Blame History

Smokes v2 — Minimal Runner and Policy

Profiles

quick

  • Purpose: Fast feedback for development
  • Target: ~45 seconds, ~400 tests (Phase 287 P2: 45.85s / 413 tests ✅)
  • Contents: Minimal gate tests - basic syntax, using resolution, essential control flow
  • Excluded: Heavy selfhost/Stage-B, crate exe, S3/LLVM integration, long-running tests (>0.4s)

integration

  • Purpose: Integration and heavier tests
  • Contents: Selfhost canaries, Stage-B, crate exe, S3/LLVM, phase-specific comprehensive tests
  • Run: ./tools/smokes/v2/run.sh --profile integration

full

  • Purpose: Complete regression coverage
  • Contents: All tests (network of quick + integration + additional)
  • Run: ./tools/smokes/v2/run.sh --profile full

plugins

  • Purpose: Plugin-specific tests (dynamic loading, etc.)
  • Run: ./tools/smokes/v2/run.sh --profile plugins

Policy

  • Use [SKIP:] prefix for environment/host dependent skips.
    • Examples: [SKIP] hakorune not built, [SKIP:env] plugin path missing
    • Keep reasons short and stable to allow grep-based canaries.
  • Prefer JSON-only output in CI: set NYASH_JSON_ONLY=1 to avoid noisy logs.
  • Diagnostics lines like [provider/select:*] are filtered by default in lib/test_runner.sh.
    • Toggle: set HAKO_SILENT_TAGS=0 to disable filtering and show raw logs. HAKO_SHOW_CALL_LOGS=1 also bypasses filtering.

Helpers

  • tools/smokes/v2/lib/mir_canary.sh provides:
    • extract_mir_from_output — between [MIR_BEGIN]/[MIR_END]
    • assert_has_tokens, assert_skip_tag, assert_order, assert_token_count
  • tools/lib/canary.sh provides minimal, harness-agnostic aliases:
    • extract_mir_between_tags — same as extract_mir_from_output
    • require_tokens token... — fail if any token missing

Notes

  • Avoid running heavy integration smokes in CI by default. Use --profile quick.
  • When a test depends on external tools (e.g., LLVM), prefer [SKIP:<reason>] over failure.
  • Stage‑B/selfhost canaries（stage1_launcher_*, phase251* など）は Stage‑3 デフォルト環境で安定しないため、quick プロファイルでは [SKIP:stageb] として扱い、必要に応じて別プロファイル（integration/full）で個別に実行する。
  • Selfhost quick カバレッジは最小 1 本（core/selfhost_minimal.sh）に絞り、Stage‑3 + JoinIR 前提で Stage‑B→VM を通るかだけを確認する。
  • S3 backend 向けの長尺テスト群も quick 向きではないため、timeout を短く保ちたい場合は [SKIP:slow] にして別途ローカルで回すことを推奨する。

Quick tips

  • EXE-heavy cases (e.g., phase2100/*) may take longer. When running quick with these tests, pass a larger timeout like --timeout 120.
  • Smokes v2 auto-cleans temporary crate EXE objects created under /tmp (pattern: ny_crate_backend_exe_*.o) after the run.

Developer Notes

  • JoinIR If/Select (Phase 33): A/B test with NYASH_FEATURES=stage3 HAKO_JOINIR_IF_SELECT=1 ./target/release/hakorune apps/tests/joinir_if_select_simple.hako（dev-only、CI対象外。NYASH_JOINIR_CORE は deprecated/無視）