Phase 22.1 WIP: SSOT resolver + TLV infrastructure + Hako MIR builder setup

Setup infrastructure for Phase 22.1 (TLV C shim & Resolver SSOT):

Core changes:
- Add nyash_tlv, nyash_c_core, nyash_kernel_min_c crates (opt-in)
- Implement SSOT resolver bridge (src/using/ssot_bridge.rs)
- Add HAKO_USING_SSOT=1 / HAKO_USING_SSOT_HAKO=1 env support
- Add HAKO_TLV_SHIM=1 infrastructure (requires --features tlv-shim)

MIR builder improvements:
- Fix using/alias consistency in Hako MIR builder
- Add hako.mir.builder.internal.{prog_scan,pattern_util} to nyash.toml
- Normalize LLVM extern calls: nyash.console.* → nyash_console_*

Smoke tests:
- Add phase2211 tests (using_ssot_hako_parity_canary_vm.sh)
- Add phase2220, phase2230, phase2231 test structure
- Add phase2100 S3 backend selector tests
- Improve test_runner.sh with quiet/timeout controls

Documentation:
- Add docs/ENV_VARS.md (Phase 22.1 env vars reference)
- Add docs/development/runtime/C_CORE_ABI.md
- Update de-rust-roadmap.md with Phase 22.x details

Tools:
- Add tools/hakorune_emit_mir.sh (Hako-first MIR emission wrapper)
- Add tools/tlv_roundtrip_smoke.sh placeholder
- Improve ny_mir_builder.sh with better backend selection

Known issues (to be fixed):
- Parser infinite loop in static method parameter parsing
- Stage-B output contamination with "RC: 0" (needs NYASH_JSON_ONLY=1)
- phase2211/using_ssot_hako_parity_canary_vm.sh fork bomb (needs recursion guard)

Next steps: Fix parser infinite loop + Stage-B quiet mode for green tests
This commit is contained in:
nyash-codex
2025-11-09 15:11:18 +09:00
parent 5d2cd5bad0
commit 981ddd890c
62 changed files with 1981 additions and 103 deletions

103
docs/ENV_VARS.md Normal file
View File

@ -0,0 +1,103 @@
# Environment Variables — Quick Reference (Phase 22.1)
This document lists the environment flags introduced or used by the Phase 22.1 work. Defaults are OFF and behavior remains unchanged unless noted.
- HAKO_USING_SSOT=0|1
- Enables the SSOT resolver gate in the runner pipeline.
- When ON, resolution first consults the SSOT bridge (modules-only MVP). If not resolved, it falls back to the existing resolver.
- Trace: set `NYASH_RESOLVE_TRACE=1` to see `[using/ssot]` tags.
- HAKO_USING_SSOT_HAKO=0|1
- Optional: within the SSOT bridge, call the Hako box `UsingResolveSSOTBox.resolve(name, ctx)` via the VM.
- MVP passes `{ modules, using_paths, cwd }` in `ctx` (modules is consulted). IO is not performed in the box.
- Requires `nyash` binary present; guard remains OFF by default.
Relative inference (SSOT)
- Default OFF: `HAKO_USING_SSOT_RELATIVE=1` enables a minimal relative candidate synthesis (cwd → using_paths). When multiple candidates exist and `NYASH_USING_STRICT=1`, resolution delegates to legacy resolver (behavior unchanged).
- Ambiguous list size: `HAKO_USING_SSOT_RELATIVE_AMBIG_FIRST_N=<N>` customizes how many candidates are shown in trace (default 3, bounded 110).
Notes on SSOT ctx (expansion plan)
- The bridge constructs a context with:
- `modules` (Map<String,String>) — exact name → path mapping
- `using_paths` (Array<String>) — resolution bases (MVP: hint only)
- `cwd` (String) — callers directory (MVP: hint only)
- Hako box will progressively leverage `using_paths`/`cwd` for relative inference (planned; defaults remain unchanged until enabled).
- HAKO_TLV_SHIM=0|1
- Enables an identity TLV roundtrip at the end of argument encoding for plugin calls.
- Requires building with `--features tlv-shim` to link the optional crate `nyash-tlv`.
- Default OFF; when OFF, the buffer is returned unchanged.
- tlv-shim (Cargo feature)
- `cargo build --features tlv-shim` links the optional `nyash-tlv` crate.
- Without this feature, `HAKO_TLV_SHIM=1` has no effect and the original path is used.
TLV shim diagnostics
- HAKO_TLV_SHIM_TRACE=0|1
- When 1 (with `tlv-shim` feature), emit a concise trace tag `[tlv/shim:<Box>.<method>]` for shimmed calls.
- Default: minimal noise`MapBox.set` のみ)。詳細な対象はフィルタで拡張。
- HAKO_TLV_SHIM_FILTER=<CommaSeparatedList>
- Filter which calls are traced例: `MapBox.set,ArrayBox.push`)。`<Box>.<method>` または `method` のみで一致。
- 未設定時は最小(`MapBox.set` のみ)。
- HAKO_TLV_SHIM_TRACE_DETAIL=0|1
- When 1, emits `[tlv/shim:detail argc=N]`.
Examples (TLV trace)
- `HAKO_TLV_SHIM=1 HAKO_TLV_SHIM_TRACE=1 HAKO_TLV_SHIM_FILTER=ArrayBox.push`
- `HAKO_TLV_SHIM=1 HAKO_TLV_SHIM_TRACE=1 HAKO_TLV_SHIM_FILTER=MapBox.get`
- `HAKO_TLV_SHIM=1 HAKO_TLV_SHIM_TRACE=1 HAKO_TLV_SHIM_FILTER=ArrayBox.push,MapBox.get` (複数指定)
- `HAKO_SHOW_CALL_LOGS=1 HAKO_CALL_TRACE=1 HAKO_CALL_TRACE_FILTER=env.console.log`テストランナーのフィルタ無効extern だけ観測)
Core Thinning I (Phase 22.2) — Plugin C wrapper (design hook)
- HAKO_PLUGIN_LOADER_C_WRAP=0|1
- When 1, emits a design-stage tag `[cwrap:invoke:<Box>.<method>]` at the plugin invocation site and then proceeds with the normal path.
- Default OFF; no behavior change.
- HAKO_PLUGIN_LOADER_C_WRAP_FILTER=<CommaSeparatedList>
- Filter for cwrap tags例: `MapBox.set,ArrayBox.push`)。`<Box>.<method>` または `method` のみで一致。
Core Thinning I (Phase 22.2) — C-core probe (design hook)
- HAKO_C_CORE_ENABLE=0|1
- When 1, emits a tag `[c-core:invoke:<Box>.<method>]` and (when built with feature `c-core`) calls a thin C probe (`nyash-c-core`) before proceeding with the normal path.
- Default OFF; behavior unchanged.
- HAKO_C_CORE_TARGETS=<CommaSeparatedList>
- Targets to probe例: `MapBox.set,ArrayBox.push`)。未設定時は `MapBox.set` のみ。
- Build note: enable C-core with `cargo build --release -p nyash-rust --features c-core`.
- Examples:
- `HAKO_C_CORE_ENABLE=1 HAKO_C_CORE_TARGETS=ArrayBox.push`
- `HAKO_C_CORE_ENABLE=1 HAKO_C_CORE_TARGETS=ArrayBox.len,ArrayBox.length`
Related toggles used by smokes/tools (for parity with runner/test wrappers):
Call/route unified trace (optional)
- HAKO_CALL_TRACE=0|1
- When ON, emits `[call:<target>.<method>]` for both plugin calls and extern calls.
- Default OFF; logs go to stderr.
- HAKO_CALL_TRACE_FILTER=<CommaSeparatedList>
- Restrict `[call:]` logs to specific targets.
- Matches `<target>.<method>` or bare `method`.
- Examples:
- `HAKO_CALL_TRACE_FILTER=MapBox.set` (method-only)
- `HAKO_CALL_TRACE_FILTER=env.console.log,MapBox.set` (mix target+method)
- HAKO_SHOW_CALL_LOGS=0|1
- When 1, test runner disables its default log filter so `[call:]` traces appear in output.
- NYASH_PARSER_STAGE3=1, HAKO_PARSER_STAGE3=1, NYASH_PARSER_ALLOW_SEMICOLON=1
- NYASH_ENTRY_ALLOW_TOPLEVEL_MAIN=1
- NYASH_DISABLE_NY_COMPILER=1, HAKO_DISABLE_NY_COMPILER=1
LLVM backend selector (builder wrapper)
- NYASH_LLVM_BACKEND=llvmlite|crate|native
- Selects the backend used by `tools/ny_mir_builder.sh` for `--emit obj|exe`.
- Default: `llvmlite` (Python harness `tools/llvmlite_harness.py`).
- `crate`: uses `./target/release/ny-llvmc` (build with `cargo build -p nyash-llvm-compiler --release`).
- `native`: reserved for future Hako-native builder.
- Linking extras for `--emit exe`: pass via `HAKO_AOT_LDFLAGS` (e.g., `-static`), `ny-llvmc` consumes `--libs`.
Name mapping note (EXE link convenience)
- nyash.console.* は C リンク時にシンボル名 `nyash_console_*` に正規化される(ドット→アンダースコア)。
- 例: `externcall nyash.console.log(i8*)` → C シンボル `nyash_console_log`
- 最小 C ランタイムPhase 22.3)の `nyash-kernel-min-c``nyash_console_log(char*)` を提供(設計段階)。
Kernel Minimal C Runtime (Phase 22.3 — design)
- NYASH_KERNEL_C_MIN=0|1
- Reserved toggle for enabling the minimal C runtime shimsdesignstage; defaults OFF
- Build: `cargo build --release -p nyash-kernel-min-c`not linked by default

View File

@ -0,0 +1,37 @@
# C Core ABI (Design Stage) — Phase 22.2
Status: design-stage shim; defaults OFF; behavior unchanged.
Purpose
- Define a minimal, stable C ABI boundary to enable future replacement of selected Rust runtime paths when Hakorune is compiled to native EXE via LLVM.
- Keep Rust path authoritative while stubbing C calls behind env+feature gates; ON/OFF parity must hold.
Conventions
- Encoding: UTF8, null-terminated (const char*). No ownership transfer.
- Return: `int` (0 = success; negative values reserved for future detailed errors).
- Threading: functions must be reentrant; stateful access goes via (type_id, instance_id).
Functions (initial)
- `int ny_core_probe_invoke(const char* target, const char* method, int32_t argc)`
- No-op probe for diagnostics; safe to call for any target/method pair.
- `int ny_core_map_set(int32_t type_id, uint32_t instance_id, const char* key, const char* val)`
- Design stub for MapBox.set. Current implementation is no-op; Rust path performs the actual mutation.
Gates & Features
- Build: `cargo build --release -p nyash-rust --features c-core`
- Env:
- `HAKO_C_CORE_ENABLE=1` — enable c-core probe routing
- `HAKO_C_CORE_TARGETS=MapBox.set,ArrayBox.push` — limit targets (default: MapBox.set)
- Tags: `[c-core:invoke:<Box>.<method>]`
Call Sites (Rust)
- PluginLoaderV2 (enabled): `src/runtime/plugin_loader_v2/enabled/ffi_bridge.rs`
- When gated ON + targeted, call C shim then continue with the original Rust path (parity preserved).
Validation
- Parity canaries compare ON/OFF outputs (and rc) for MapBox.set; later for ArrayBox.push/get/size.
- Failure/unavailable paths must fall back immediately to the Rust path.
Roadmap
- Expand ArrayBox (push → get → size) with the same staged approach.
- Formalize error codes and minimal state API only after parity is stable.

View File

@ -10,41 +10,56 @@ Purpose: reduce Rust surface (nonplugin) while keeping correctness and revers
- Acceptance: `cargo build --release` (default features) remains green; quick smokes green.
- Revert: `git mv archive/rust-llvm-backend/llvm src/backend/`.
### Phase 1 — Quick Wins (12 months)
- hv1_inline → Hako parity (already functionally covered; keep as perf path).
- MIR interpreter parity check; route Primary to Hako VM.
- TLV codec → C shim (FFI) with thin Rust wrapper.
- LLVM wrapper → Hako/C harness (Python stays primary until Hako IR is ready).
### Phase 1 — Parser/MIR Hakofirst22.0
- Make Hako the primary for Parser/MIR; Rust builder becomes fallback.
- Verify quick canaries green under registry; keep hv1 inline parity.
### Phase 2 — TLV C shim & Resolver SSOT22.1
- TLV codec to C shim (+ Rust FFI); Resolver/Using SSOT in Hako shared by runner/analyzer.
Deliverables & Tests
- hv1_inline: parity canaries = phase2037/flow + phase2170/state; add `HAKO_VERIFY_DISABLE_INLINE` opt-out (optional).
- TLV C shim: roundtrip tests (encode→decode) on representative payloads; FFI error mapping spec.
- MIR interpreter: keep as diagnostic path; parity sample set (const/binop/compare/branch/jump/ret/phi/mir_call minimal).
- LLVM wrapper: unify CLI entry; ensure `NYASH_LLVM_USE_HARNESS=1` path stays green.
- 22.0: registry builder default ON; hv1直列 green; Core/Interpreter is diagnostic.
- 22.1: TLV roundtrip smokes; Using SSOT parity between runner/analyzer.
### Phase 2 — Core Thinning (24 months)
- Plugin loader thin C wrapper (dlopen/dlsym), unify host ABI.
- Basic boxes (Array/Map/File) small C core (handlebased), keep Rust shim.
- Resolver/Using: SSOT in Hako; runner uses shared policy.
### Phase 3 — Core Thinning I22.2, 24 months
- Plugin loader thin C wrapper (dlopen/dlsym) and basic boxes C core; Rust shim remains.
SSOT for Using/Resolver (summary)
- Resolution order: modules (nyash.toml) → relative path inference → not found (warn) with verbose details.
- Analyzer/HakoCheck follows the same order; runner shares policy helpers. No pathliteral using in strict profiles.
### Phase 3 — Longhaul (36 months)
### Phase 4 — Longhaul22.3, 36 months
- Python llvmlite → Hako IR builder + C ABI.
- Parser/MIR builder fully Hakofirst; Rust becomes fallback.
- NyKernel minimal C runtime (BoxCall dispatcher + collections + file).
### Phase 21.10 — LLVM Line Unification (SSOT + crate probe)
- SSOT builder (`tools/ny_mir_builder.sh`) selects backend by env; crate path optin.
- Add crate S3 canaries (ternary/map/print); defaults unchanged.
### Phase 21.11 — Flip default to crate (ny-llvmc)
- Make crate default when available; llvmlite becomes optin.
- S3 reps run via crate in quick; legacy remains available.
### Phase 21.12 — Hako Native LLVM Builder (bootstrap)
- Experimental native (Hako→LLVM C API) path for minimal EXE.
- Behind `NYASH_LLVM_BACKEND=native` toggle; no default impact.
### Phase 21.13 — llvmlite deprecation (default off)
- Remove llvmlite from auto paths; keep explicit toggle + optional CI job.
### Phase 21.14 — Optimization & Perf Harness
- Perf harness + PHI invariants; optimize hot paths; publish numbers.
## Principles
- Guard everything by env/features; defaults unchanged.
- Keep changes reversible (small diffs, RESTORE docs, fallbacks).
- Test gates: quick smokes + representative hv1/hakovm parity.
## Today (suggested)
1) Stage Phase0 move as a script (not autorun) + RESTORE.md.
2) Add phase docs (docs/private/roadmap/phases/phase-21.9/PLAN.md).
3) Keep CI/dev instructions intact (no build break when features=none).
1) Lock 22.0 (Parser/MIR Hakofirst) — builder registryを既定ON、quickが緑。
2) Prepare 22.1 (TLV C shim & Resolver SSOT) — I/F草案と最小スモーク。
3) LLVM統一21.1021.14は並行で準備、切替は22.x完了後に本格実施。
## Test Strategy (gates)
- Quick: tools/smokes/v2/profiles/quick/core/* (phase2037 flow, phase2170 state) — green.