diff --git a/CLAUDE.md b/CLAUDE.md index 145a3bc4..f98e29ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -66,8 +66,11 @@ NYASH_CLI_VERBOSE=1 ./tools/jit_smoke.sh # ラウンドトリップ (パーサーパイプ + JSON) ./tools/ny_roundtrip_smoke.sh -# Nyコンパイラ MVP経路 (実験的) +# Nyコンパイラ MVP経路 (Phase 15.3実装中!) NYASH_USE_NY_COMPILER=1 ./target/release/nyash program.nyash + +# JSON v0 Bridge経由実行(完成済み) +python tools/ny_parser_mvp.py program.nyash | ./target/release/nyash --ny-parser-pipe ``` ### 🐧 Linux/WSL版 @@ -193,6 +196,12 @@ NYASH_DISABLE_PLUGINS=1 ./target/release/nyash program.nyash # ラウンドトリップテスト ./tools/ny_roundtrip_smoke.sh +# Stage-2 PHIスモーク(If/Loop PHI合流) +./tools/ny_parser_stage2_phi_smoke.sh + +# Stage-2 Bridgeスモーク(算術/比較/短絡/if) +./tools/ny_stage2_bridge_smoke.sh + # プラグインスモーク(オプション) NYASH_SKIP_TOML_ENV=1 ./tools/smoke_plugins.sh @@ -223,10 +232,13 @@ NYASH_LLVM_USE_HARNESS=1 ./target/release/nyash program.nyash ## 📝 Update (2025-09-14) 🎉 セルフホスティング大前進! - ✅ Python LLVM実装が実用レベル到達!(esc_dirname_smoke, min_str_cat_loop, dep_tree_min_string全てPASS) -- 🚀 パーサーのNyash実装開始!ChatGPT5が`apps/selfhost/parser/`で実装中 +- 🚀 **Phase 15.3開始!** NyashコンパイラMVP実装が`apps/selfhost-compiler/`でスタート! +- ✅ JSON v0 Bridge完成 - If/Loop PHI生成実装済み(ChatGPT実装) +- 🔧 Python MVPパーサーStage-2完成 - local/if/loop/call/method/new対応 - 📚 peek式の再発見 - when→peekに名前変更、ブロック/値/文すべて対応済み - 🧠 箱理論でSSA構築を簡略化(650行→100行)- 論文執筆完了 - 🤝 AI協働の知見を論文化 - 実装駆動型学習の重要性を実証 +- 🎯 **LoopForm戦略決定**: PHIは逆Lowering時に自動生成(Codex推奨) - 📋 詳細: [Phase 15 README](docs/development/roadmap/phases/phase-15/README.md) ### 🚀 新発見:プラグイン全方向ビルド戦略 @@ -246,6 +258,12 @@ clang main.o filebox.o pathbox.o libnyrt.a -o nyash_static.exe ### 🏗️ Everything is Box - すべての値がBox(StringBox, IntegerBox, BoolBox等) - ユーザー定義Box: `box ClassName { field1: TypeBox field2: TypeBox }` +- **MIR14命令**: たった14個の命令で全機能実現! + - 基本演算(5): Const, UnaryOp, BinOp, Compare, TypeOp + - メモリ(2): Load, Store + - 制御(4): Branch, Jump, Return, Phi + - Box(2): NewBox, BoxCall + - 外部(1): ExternCall ### 🌟 完全明示デリゲーション ```nyash diff --git a/CURRENT_TASK.md b/CURRENT_TASK.md index 679cacf0..0ba711b3 100644 --- a/CURRENT_TASK.md +++ b/CURRENT_TASK.md @@ -1,12 +1,39 @@ # Current Task (2025-09-14 改定) — Phase 15 llvmlite(既定)+ PyVM(新規) +Handoff — TL;DR(2025‑09‑15) +- フェーズ: 15.3(Ny コンパイラMVP 入口を統合済み/Stage‑1 実装中) +- ランナー統合: `NYASH_USE_NY_COMPILER=1` で selfhost compiler を子プロセス実行→stdout の JSON v0 を Bridge→MIR 実行(失敗時は自動フォールバック) +- Selfhost compiler 配置: `apps/selfhost-compiler/`(`compiler.nyash` がエントリ。parser/emitter骨子あり) +- 進捗: + - Stage‑1 パーサ骨子(number/string/term/expr/returnスキップ)実装済み(2パス gpos 方針)。 + - セミコロン最小対応: selfhost 側で `;` を空白同等にスキップ(改行ベース+必要時のみ;)。Rustパーサは未対応のまま。 + - 仕様ドキュメント更新: `docs/reference/language/statements.md`(改行+最小ASI)。 + - imports/namespace/nyash.toml: Phase‑15 中は Runner(Rust) で解決継続。selfhost は `using` 受理(no‑op)を 15.3後半に導入予定(計画: `docs/development/roadmap/phases/phase-15/imports-namespace-plan.md`)。VM 変更は不要。 +- そのまま回せるスモーク(緑): + - `tools/ny_stage2_bridge_smoke.sh`(算術/比較/短絡/ネストif) + - `tools/ny_parser_stage2_phi_smoke.sh`(If/Loop PHI) + - `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/esc_dirname_smoke.nyash` + - `tools/selfhost_compiler_smoke.sh`(selfhost 入口/MVPはResult:0) +- 次にやること(短期): + 1) Stage‑1: `return 1+2*3` の JSON v0 正常化(左結合/優先度準拠)→ Bridge → MIR 実行 = 7 を確定。 + 2) 最小ASI導入(深さ0・直前が継続子でない改行のみ文終端)。代表ケースのスモーク追加(if/else連結、return継続、演算子継続、ドットチェイン)。 + 3) Stage‑2 着手(local/if/loop/call/method/new/var/論理/比較)— PHI は Bridge 側に委譲のまま。 +- 守ること: + - PHI: Bridge 側(If/Loop)で合流、LoopForm(Core‑14) 導入時に逆Loweringで自動化へ(現状維持)。 + - imports/namespace: 15.3 は Runner で解決維持(selfhost は no‑op 受理のみ)。 + - フォールバックは常時安全(selfhost 経路失敗→既存経路)。 + + Quick Summary(反映内容) - 実装済み: peek CFG修正、llvmlite文字列ブリッジ、混在‘+’結合、追加テスト緑。 - JSON v0 Bridge拡張: Stmt(Expr/Local/If/Loop)、Expr(Call/Method/New/Var) 降下を実装。 - PHI合流: If/Loop の PHI 合流を Bridge 側で実装(If: then/else→merge、Loop: preheader/loop‑latch→header)。 - PythonパーサMVP: Stage‑2 サブセット(local/if/loop/call/method/new/var ほか)を出力。 +- Ny コンパイラMVP(入口): `NYASH_USE_NY_COMPILER=1` で Ny→JSON v0 パスを試行(失敗時は自動フォールバック)。初期実装は `apps/selfhost-compiler/compiler.nyash`(優先)/`apps/selfhost/parser/ny_parser_v0/main.nyash` を子プロセスで実行して JSON を収集。 +- 文分離ポリシー: 「改行ベース+必要時のみセミコロン」。最小ASI(深さ0かつ直前が継続子でない改行のみ文終端)。ドキュメント反映済み。 - Outstanding: then/else 片側のみで新規生成された変数のスコープ(現在は外へ未伝播)。 - Next(handoff): Stage‑2 E2E緑化→me の扱い検討(Method糖衣)。 +- 後続計画: MIR‑SSA(Sealed SSA)骨子を 15.2 終盤で投入(既定OFF)、15.3 で並走→15.3 終盤で既定化。 - How to run: 下の手順に確認コマンドを追記。 Hot Update — 2025‑09‑14(JSON v0 Bridge/Parser Stage‑2 + Parity hardening) @@ -15,6 +42,7 @@ Hot Update — 2025‑09‑14(JSON v0 Bridge/Parser Stage‑2 + Parity hardeni - console.* の文字列引数を to_i8p_h ブリッジで正規化(from_i8_string 直呼び回避)。 - “文字列+数値” 混在 ‘+’ を concat_si/is+from_i8_string による橋渡しで安定化、両辺文字列は concat_hh。 - 代表追加テスト(緑): string_ops_basic / me_method_call / loop_if_phi。 + - 追加パリティ確認(緑): string_ops_basic / peek_expr_block / peek_return_value / ternary_basic / ternary_nested / esc_dirname_smoke。 - JSON v0 Bridge(Option A)の受け口を Stage‑2 方向に拡張(src/runner/json_v0_bridge.rs) - StmtV0: Expr / Local / If / Loop を追加。If/Loop は実ブロック生成(then/else/merge、cond/body/exit)まで実装、未終端 Jump 補完、最後に未終端ブロックへ ret 0 補完。 @@ -35,7 +63,10 @@ Outstanding(要対応) Next(handoff short plan) 1) JSON Bridge: PHI 合流実装(If/Loop)と最小テスト追加(ループ後/if後の変数参照)。 2) Parser MVP: Stage‑2 生成 JSON の if/loop ケースで Bridge→MIR→PyVM/llvmlite の parity 緑化。 -3) me の扱い検討(当面は Method 降下で十分。必要なら me→Main.method/N 直呼シンタックスを Bridge 側で糖衣対応)。 +3) me の扱い(MVP方針) + - JSON v0: `Method{ recv: Var{"me"} }` を許容(文脈があれば var_map に解決)。 + - Bridge: 既定は未定義エラー。デバッグ用に `NYASH_BRIDGE_ME_DUMMY=1`(任意 `NYASH_BRIDGE_ME_CLASS`=Main 既定)で `NewBox{class}` を注入して `me` を仮解決。 + - 将来: Ny Builder が box/method 文脈で `me` を提供(Bridge のダミーは撤去予定)。 How to run(現状確認) - Build(release): `cargo build --release`(必要に応じて `NYASH_CLI_VERBOSE=1`)。 @@ -44,6 +75,38 @@ How to run(現状確認) - Parser Stage‑2 → Bridge: `python3 tools/ny_parser_mvp.py tmp/sample.ny | target/release/nyash --ny-parser-pipe`。 - Bridge smoke(一式): `./tools/ny_parser_bridge_smoke.sh`(pipe/--json-file の両経路)。 - Stage‑2 PHI smoke: `./tools/ny_parser_stage2_phi_smoke.sh`(If/Loop の PHI 合流検証)。 +- me dummy smoke(デバッグゲート): `./tools/ny_me_dummy_smoke.sh`(`NYASH_BRIDGE_ME_DUMMY=1` で Var("me") をダミー注入)。 +- Stage‑2 Bridge smoke(サブセット一括): `./tools/ny_stage2_bridge_smoke.sh`(算術/比較/短絡/ネストif)。 +- 自己ホスト準備(ブートストラップ): `./tools/bootstrap_selfhost_smoke.sh`(c0→c1→c1'、MVPはフォールバック許容)。 +- Nyコンパイラ入口スモーク: `./tools/selfhost_compiler_smoke.sh`(json_v0 emit → Result:0)。 + +Phase 15.3 — Ny compiler MVP 詳細計画(抜粋) +- 入口: Runner に `NYASH_USE_NY_COMPILER=1` を追加し、selfhost compiler を子プロセス実行(stdout の JSON v0 を Bridge へ)。失敗は自動フォールバック。 +- Stage‑1(小さく積む) + 1) return / 整数 / 文字列 / 四則 / 括弧(左結合)。 + 2) 文分離(最小ASI): 改行=文区切り、継続子(+ - * / . ,)やグルーピング中は継続。 + 3) スモーク: `return 1+2*3` → JSON v0 → Bridge → MIR 実行 = 7。 +- Stage‑2(本命) + - local / if / loop / call / method / new / var / 比較 / 論理(短絡)。 + - PHI: Bridge 側の合流(If/Loop)に依存(Phase‑15 中は現行維持)。 + - スモーク: nested if / loop 累積 / 短絡 and/or × if/loop。 +- 受け入れ(15.3) + - Stage‑1: 代表サンプル緑(PyVM/llvmlite 一致)。 + - Bootstrap: `tools/bootstrap_selfhost_smoke.sh` PASS(フォールバックは許容)。 + - Docs: `docs/reference/language/statements.md` 公開(方針/例/実装ノート)。 + +Parity memo(確認済み) +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/string_ops_basic.nyash` → 緑 +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/peek_expr_block.nyash` → 緑 +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/peek_return_value.nyash` → 緑 +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/ternary_basic.nyash` → 緑 +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/ternary_nested.nyash` → 緑 +- `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/esc_dirname_smoke.nyash` → 緑 + +Notes — PHI 方針(Phase‑15→Core‑14) +- Phase‑15: 現行の Bridge‑PHI(If/Loop 合流)を維持して完成まで駆け抜ける(変更しない)。 +- Core‑14: LoopForm を強化し、逆Loweringで PHI を自動生成(構造ブロック→合流点 PHI)。 +- JSON v0: Phase‑15 は従来通り PHI を含める/Core‑14 以降は非必須(将来は除外方向)。 Context Snapshot — Open After Reset - Status: A6 受入(PyVM↔llvmlite parity + LLVM verify→.o→EXE)完了。 @@ -120,6 +183,7 @@ Hot Update — 2025‑09‑14(Language + Docs) - 言語索引: `docs/reference/language/README.md` - 安定パス(スタブ): `docs/reference/language/LANGUAGE_REFERENCE_2025.md`(private実体へ誘導) - アーキ索引/受け皿: `docs/reference/architecture/{nyash_core_concepts.md, execution-backends.md, TECHNICAL_ARCHITECTURE_2025.md}` + - 文分離ポリシー: `docs/reference/language/statements.md`(改行ベース+必要時のみセミコロン、最小ASI) - Parser MVP(Stage 1): - Python 実装: `tools/ny_parser_mvp.py` を追加、Roundtrip スモーク `tools/ny_parser_mvp_roundtrip.sh` で緑。 - Nyash 実装スケルトン: `apps/selfhost/parser/ny_parser_v0/main.nyash`(改修継続)。 diff --git a/app_parity_esc_dirname_smoke b/app_parity_esc_dirname_smoke index 4838e9ad..6dd284d8 100644 Binary files a/app_parity_esc_dirname_smoke and b/app_parity_esc_dirname_smoke differ diff --git a/app_parity_ternary_basic b/app_parity_ternary_basic index a8ddf97f..bd11e67a 100644 Binary files a/app_parity_ternary_basic and b/app_parity_ternary_basic differ diff --git a/app_parity_ternary_nested b/app_parity_ternary_nested index ee7d8dca..40f13645 100644 Binary files a/app_parity_ternary_nested and b/app_parity_ternary_nested differ diff --git a/apps/selfhost-compiler/README.md b/apps/selfhost-compiler/README.md new file mode 100644 index 00000000..e6dd8569 --- /dev/null +++ b/apps/selfhost-compiler/README.md @@ -0,0 +1,18 @@ +# Nyash Selfhost Compiler (MVP scaffold) + +This is the Phase 15.3 work-in-progress Nyash compiler implemented in Ny. + +Layout +- `compiler.nyash`: entry (CompilerBox). Reads `tmp/ny_parser_input.ny`, prints JSON v0. +- `parser/`: lexer/parser/ast (scaffolds; to be filled as we extend Stage‑2) +- `mir/`: builder/optimizer stubs (future; current target is JSON v0 emit) +- `tests/`: Stage‑1/2 samples (TBD) + +Run (behind flag) +- `NYASH_USE_NY_COMPILER=1 target/release/nyash --backend vm ` + - The runner writes the input to `tmp/ny_parser_input.ny` and invokes this program. + - It captures a JSON v0 line from stdout and executes it via the JSON bridge. + +Notes +- Early MVP emits a minimal JSON v0 (currently a placeholder: return 0). We will gradually wire lexer/parser/emitter. +- Keep JSON v0 spec in `docs/reference/ir/json_v0.md`. diff --git a/apps/selfhost-compiler/compiler.nyash b/apps/selfhost-compiler/compiler.nyash new file mode 100644 index 00000000..111e2650 --- /dev/null +++ b/apps/selfhost-compiler/compiler.nyash @@ -0,0 +1,309 @@ +// Selfhost Compiler MVP (Phase 15.3) +// Reads tmp/ny_parser_input.ny and prints a minimal JSON v0 program. + +static box Main { + // ---- IO helper ---- + read_all(path) { + local fb = new FileBox() + fb.open(path, "r") + local s = fb.read() + fb.close() + return s + } + + // ---- JSON helpers ---- + esc_json(s) { + local out = "" + local i = 0 + local n = s.length() + loop(i < n) { + local ch = s.substring(i, i+1) + if ch == "\\" { out = out + "\\\\" } else { + if ch == "\"" { out = out + "\\\"" } else { out = out + ch } + } + i = i + 1 + } + return out + } + + // ---- Lexer helpers ---- + is_digit(ch) { + return ch == "0" || ch == "1" || ch == "2" || ch == "3" || ch == "4" || ch == "5" || ch == "6" || ch == "7" || ch == "8" || ch == "9" + } + is_space(ch) { return ch == " " || ch == "\t" || ch == "\n" || ch == "\r" || ch == ";" } + + // ---- Parser (Stage-1/mini Stage-2) ---- + // Global cursor for second-pass parser (no pack strings) + gpos_set(i) { + me.gpos = i + return 0 + } + gpos_get() { return me.gpos } + + parse_number2(src, i) { + local n = src.length() + local j = i + loop(j < n && me.is_digit(src.substring(j, j+1))) { j = j + 1 } + local s = src.substring(i, j) + me.gpos_set(j) + return "{\"type\":\"Int\",\"value\":" + s + "}" + } + + parse_string2(src, i) { + local n = src.length() + local j = i + 1 + local out = "" + local done = 0 + loop(j < n && done == 0) { + local ch = src.substring(j, j+1) + if ch == "\"" { + j = j + 1 + done = 1 + } else { + if ch == "\\" && j + 1 < n { + local nx = src.substring(j+1, j+2) + if nx == "\"" { out = out + "\"" } else { if nx == "\\" { out = out + "\\" } else { out = out + nx } } + j = j + 2 + } else { + out = out + ch + j = j + 1 + } + } + } + me.gpos_set(j) + return "{\"type\":\"Str\",\"value\":\"" + me.esc_json(out) + "\"}" + } + + parse_factor2(src, i) { + local j = me.skip_ws(src, i) + local ch = src.substring(j, j+1) + if ch == "(" { + local inner = me.parse_expr2(src, j + 1) + local k = me.gpos_get() + k = me.skip_ws(src, k) + if src.substring(k, k+1) == ")" { k = k + 1 } + me.gpos_set(k) + return inner + } + if ch == "\"" { return me.parse_string2(src, j) } + return me.parse_number2(src, j) + } + + parse_term2(src, i) { + local lhs = me.parse_factor2(src, i) + local j = me.gpos_get() + local cont = 1 + loop(cont == 1) { + j = me.skip_ws(src, j) + if j >= src.length() { cont = 0 } else { + local op = src.substring(j, j+1) + if op != "*" && op != "/" { cont = 0 } else { + local rhs = me.parse_factor2(src, j+1) + j = me.gpos_get() + lhs = "{\"type\":\"Binary\",\"op\":\"" + op + "\",\"lhs\":" + lhs + ",\"rhs\":" + rhs + "}" + } + } + } + me.gpos_set(j) + return lhs + } + + parse_expr2(src, i) { + local lhs = me.parse_term2(src, i) + local j = me.gpos_get() + local cont = 1 + loop(cont == 1) { + j = me.skip_ws(src, j) + if j >= src.length() { cont = 0 } else { + local op = src.substring(j, j+1) + if op != "+" && op != "-" { cont = 0 } else { + local rhs = me.parse_term2(src, j+1) + j = me.gpos_get() + lhs = "{\"type\":\"Binary\",\"op\":\"" + op + "\",\"lhs\":" + lhs + ",\"rhs\":" + rhs + "}" + } + } + } + me.gpos_set(j) + return lhs + } + + parse_program2(src) { + local i = me.skip_ws(src, 0) + local j = me.skip_return_kw(src, i) + if j == i { j = i } // optional 'return' + local expr = me.parse_expr2(src, j) + return "{\"version\":0,\"kind\":\"Program\",\"body\":[{\"type\":\"Return\",\"expr\":" + expr + "}]}" + } + i2s(n) { + // integer to decimal string (non-negative only for MVP) + if n == 0 { return "0" } + local x = n + if x < 0 { x = 0 } // MVP: clamp negatives to 0 to avoid surprises + local out = "" + loop(x > 0) { + local q = x / 10 + local d = x - q * 10 + local ch = "0" + if d == 1 { ch = "1" } else { + if d == 2 { ch = "2" } else { + if d == 3 { ch = "3" } else { + if d == 4 { ch = "4" } else { + if d == 5 { ch = "5" } else { + if d == 6 { ch = "6" } else { + if d == 7 { ch = "7" } else { + if d == 8 { ch = "8" } else { + if d == 9 { ch = "9" } + } + } + } + } + } + } + } + } + out = ch + out + x = q + } + return out + } + to_int(s) { + local n = s.length() + if n == 0 { return 0 } + local i = 0 + local acc = 0 + loop(i < n) { + local d = s.substring(i, i+1) + local dv = 0 + if d == "1" { dv = 1 } else { if d == "2" { dv = 2 } else { if d == "3" { dv = 3 } else { if d == "4" { dv = 4 } else { if d == "5" { dv = 5 } else { if d == "6" { dv = 6 } else { if d == "7" { dv = 7 } else { if d == "8" { dv = 8 } else { if d == "9" { dv = 9 } } } } } } } } } + acc = acc * 10 + dv + i = i + 1 + } + return acc + } + + parse_number(src, i) { + local n = src.length() + local j = i + loop(j < n && me.is_digit(src.substring(j, j+1))) { j = j + 1 } + local s = src.substring(i, j) + local json = "{\"type\":\"Int\",\"value\":" + s + "}" + return json + "@" + me.i2s(j) + } + + parse_string(src, i) { + local n = src.length() + local j = i + 1 + local out = "" + loop(j < n) { + local ch = src.substring(j, j+1) + if ch == "\"" { + j = j + 1 + return "{\"type\":\"Str\",\"value\":\"" + me.esc_json(out) + "\"}@" + j + } + if ch == "\\" && j + 1 < n { + local nx = src.substring(j+1, j+2) + if nx == "\"" { out = out + "\"" } else { if nx == "\\" { out = out + "\\" } else { out = out + nx } } + j = j + 2 + } else { + out = out + ch + j = j + 1 + } + } + return "{\"type\":\"Str\",\"value\":\"" + me.esc_json(out) + "\"}@" + me.i2s(j) + } + + skip_ws(src, i) { + local n = src.length() + loop(i < n && me.is_space(src.substring(i, i+1))) { i = i + 1 } + return i + } + + skip_return_kw(src, i) { + // If source at i starts with "return", advance; otherwise return i unchanged + local n = src.length() + local j = i + if j < n && src.substring(j, j+1) == "r" { j = j + 1 } else { return i } + if j < n && src.substring(j, j+1) == "e" { j = j + 1 } else { return i } + if j < n && src.substring(j, j+1) == "t" { j = j + 1 } else { return i } + if j < n && src.substring(j, j+1) == "u" { j = j + 1 } else { return i } + if j < n && src.substring(j, j+1) == "r" { j = j + 1 } else { return i } + if j < n && src.substring(j, j+1) == "n" { j = j + 1 } else { return i } + return j + } + + parse_factor(src, i) { + i = me.skip_ws(src, i) + local ch = src.substring(i, i+1) + if ch == "(" { + local p = me.parse_expr(src, i + 1) + local at = p.lastIndexOf("@") + local ej = p.substring(0, at) + local j = me.to_int(p.substring(at+1, p.length())) + j = me.skip_ws(src, j) + if src.substring(j, j+1) == ")" { j = j + 1 } + return ej + "@" + me.i2s(j) + } + if ch == "\"" { return me.parse_string(src, i) } + return me.parse_number(src, i) + } + + parse_term(src, i) { + local p = me.parse_factor(src, i) + local at = p.lastIndexOf("@") + local lhs = p.substring(0, at) + local j = me.to_int(p.substring(at+1, p.length())) + local cont = 1 + loop(cont == 1) { + j = me.skip_ws(src, j) + if j >= src.length() { cont = 0 } else { + local op = src.substring(j, j+1) + if op != "*" && op != "/" { cont = 0 } else { + local q = me.parse_factor(src, j+1) + local at2 = q.lastIndexOf("@") + local rhs = q.substring(0, at2) + j = me.to_int(q.substring(at2+1, q.length())) + lhs = "{\"type\":\"Binary\",\"op\":\"" + op + "\",\"lhs\":" + lhs + ",\"rhs\":" + rhs + "}" + } + } + } + return lhs + "@" + me.i2s(j) + } + + parse_expr(src, i) { + local p = me.parse_term(src, i) + local at = p.lastIndexOf("@") + local lhs = p.substring(0, at) + local j = me.to_int(p.substring(at+1, p.length())) + local cont = 1 + loop(cont == 1) { + j = me.skip_ws(src, j) + if j >= src.length() { cont = 0 } else { + local op = src.substring(j, j+1) + if op != "+" && op != "-" { cont = 0 } else { + local q = me.parse_term(src, j+1) + local at2 = q.lastIndexOf("@") + local rhs = q.substring(0, at2) + j = me.to_int(q.substring(at2+1, q.length())) + lhs = "{\"type\":\"Binary\",\"op\":\"" + op + "\",\"lhs\":" + lhs + ",\"rhs\":" + rhs + "}" + } + } + } + return lhs + "@" + me.i2s(j) + } + + parse_program(src) { + // Legacy packed path (debug) removed; use parser2 + return me.parse_program2(src) + } + + main(args) { + // Parse the input and emit JSON v0 + local src = me.read_all("tmp/ny_parser_input.ny") + if src == null { src = "return 1+2*3" } + local json = me.parse_program(src) + local console = new ConsoleBox() + // console.println(json) -- final output only + console.println(json) + return 0 + } +} diff --git a/apps/selfhost-compiler/emitter/json_v0.nyash b/apps/selfhost-compiler/emitter/json_v0.nyash new file mode 100644 index 00000000..4dab9a6a --- /dev/null +++ b/apps/selfhost-compiler/emitter/json_v0.nyash @@ -0,0 +1,8 @@ +// JSON v0 emitter (MVP placeholder) +static box JsonV0Emitter { + // Emit a minimal Program{return 0} + program_return0() { + return "{\"version\":0,\"kind\":\"Program\",\"body\":[{\"type\":\"Return\",\"expr\":{\"type\":\"Int\",\"value\":0}}]}" + } +} + diff --git a/apps/selfhost-compiler/mir/builder.nyash b/apps/selfhost-compiler/mir/builder.nyash new file mode 100644 index 00000000..16324230 --- /dev/null +++ b/apps/selfhost-compiler/mir/builder.nyash @@ -0,0 +1,5 @@ +static box MirBuilder { + // placeholder + main(args) { return 0 } +} + diff --git a/apps/selfhost-compiler/mir/optimizer.nyash b/apps/selfhost-compiler/mir/optimizer.nyash new file mode 100644 index 00000000..4df27d1f --- /dev/null +++ b/apps/selfhost-compiler/mir/optimizer.nyash @@ -0,0 +1,5 @@ +static box Optimizer { + // placeholder + main(args) { return 0 } +} + diff --git a/apps/selfhost-compiler/parser/ast.nyash b/apps/selfhost-compiler/parser/ast.nyash new file mode 100644 index 00000000..912dd3af --- /dev/null +++ b/apps/selfhost-compiler/parser/ast.nyash @@ -0,0 +1,5 @@ +static box AST { + // scaffold for future AST node constructors + main(args) { return 0 } +} + diff --git a/apps/selfhost-compiler/parser/lexer.nyash b/apps/selfhost-compiler/parser/lexer.nyash new file mode 100644 index 00000000..06051d74 --- /dev/null +++ b/apps/selfhost-compiler/parser/lexer.nyash @@ -0,0 +1,5 @@ +static box Lexer { + // scaffold for future implementation + main(args) { return 0 } +} + diff --git a/apps/selfhost-compiler/parser/parser.nyash b/apps/selfhost-compiler/parser/parser.nyash new file mode 100644 index 00000000..70193cf8 --- /dev/null +++ b/apps/selfhost-compiler/parser/parser.nyash @@ -0,0 +1,5 @@ +static box Parser { + // scaffold for future implementation + main(args) { return 0 } +} + diff --git a/apps/selfhost-compiler/tests/stage1/README.md b/apps/selfhost-compiler/tests/stage1/README.md new file mode 100644 index 00000000..85a037a6 --- /dev/null +++ b/apps/selfhost-compiler/tests/stage1/README.md @@ -0,0 +1,4 @@ +Stage‑1 tests (scaffold) + +Add minimal Ny source samples here. Harness TBD. + diff --git a/docs/development/roadmap/phases/phase-15/README.md b/docs/development/roadmap/phases/phase-15/README.md index 682aa355..b6445d49 100644 --- a/docs/development/roadmap/phases/phase-15/README.md +++ b/docs/development/roadmap/phases/phase-15/README.md @@ -31,6 +31,14 @@ MIR 13命令の美しさを最大限に活かし、外部コンパイラ依存 - dep_tree_min_string: PyVM↔llvmlite パリティ一致、llvmlite 経路で `.ll verify → .o → EXE` 完走。 - 一時救済ゲート `NYASH_LLVM_ESC_JSON_FIX` は受入には未使用(OFF)。 +#### PHI 取り扱い方針(Phase‑15 中) +- 現行: JSON v0 Bridge 側で If/Loop の PHI を生成(安定・緑)。 +- 方針: Phase‑15 ではこのまま完成させる(変更しない)。 +- 理由: LoopForm(Core‑14)導入時に、逆Loweringで PHI を自動生成する案(推薦)に寄せるため。 + - PHI は「合流点での別名付け」であり、Boxの操作ではない。 + - 抽象レイヤの純度維持(Everything is Box)。 + - 実装責務の一極化(行数削減/保守性向上)。 + ### Phase 15.3: NyashコンパイラMVP(次フェーズ着手) - PyVM 安定後、Nyash製パーサ/レクサ(サブセット)と MIR ビルダを段階導入 - フラグでRustフォールバックと併存(例: `NYASH_USE_NY_COMPILER=1`) @@ -41,16 +49,76 @@ MIR 13命令の美しさを最大限に活かし、外部コンパイラ依存 - ステージ2: 文/式サブセット拡張(local/if/loop/call/method/new/me/substring/length/lastIndexOf)。 - ステージ3: Ny AST→MIR JSON 降下(直接 llvmlite/PyVM へ渡す)。 +#### Phase 15.3 — Detailed Plan(Ny compiler MVP) +- Directory layout(selfhost compiler) + - `apps/selfhost-compiler/compiler.nyash`(CompilerBox entry; Ny→JSON v0 emit) + - `apps/selfhost-compiler/parser/{lexer.nyash,parser.nyash,ast.nyash}`(Stage‑2 へ段階拡張) + - `apps/selfhost-compiler/emitter/json_v0.nyash`(将来: emit 分離。MVPは inline でも可) + - `apps/selfhost-compiler/mir/{builder.nyash,optimizer.nyash}`(将来) + - `apps/selfhost-compiler/tests/{stage1,stage2}`(サンプルと期待JSON) + +- Runner integration(安全ゲート) + - フラグ: `NYASH_USE_NY_COMPILER=1`(既定OFF) + - 子プロセス: `--backend vm` で selfhost compiler を起動し、stdout から JSON v0 1行を収集 + - 環境: `NYASH_JSON_ONLY=1` を子に渡して余計な出力を抑制。失敗時は静かにフォールバック + +- Stage‑1(小さく積む) + 1) return / 整数 / 文字列 / 四則 / 括弧(左結合) + 2) 文分離(最小ASI): 改行=文区切り、継続子(+ - * / . ,)やグルーピング中は継続 + 3) 代表スモーク: `return 1+2*3` → JSON v0 → Bridge → MIR 実行 = 7 + +- Stage‑2(本命へ) + - local / if / loop / call / method / new / var / 比較 / 論理(短絡) + - PHI: Bridge 側の合流(If/Loop)に依存(Phase‑15中は現行維持) + - 代表スモーク: nested if / loop 累積 / 短絡 and/or と if/loop の交錯 + +- Acceptance(15.3) + - Stage‑1: 代表サンプルで JSON v0 emit → Bridge → PyVM/llvmlite で一致(差分なし) + - Bootstrap: `tools/bootstrap_selfhost_smoke.sh` で c0→c1→c1' が PASS(フォールバックは許容) + - Docs: 文分離ポリシー(改行+最小ASI)を公開(link: reference/language/statements.md) + +- Smokes / Tools(更新) + - `tools/selfhost_compiler_smoke.sh`(入口) + - `tools/ny_stage2_bridge_smoke.sh`(算術/比較/短絡/ネストif) + - `tools/ny_parser_stage2_phi_smoke.sh`(If/Loop の PHI 合流) + - `tools/parity.sh --lhs pyvm --rhs llvmlite `(常時) + +Imports/Namespace plan(15.3‑late) +- See: imports-namespace-plan.md — keep `nyash.toml` resolution in runner; accept `using` in Ny compiler as no‑op (no resolution) gated by `NYASH_ENABLE_USING=1`. + +- Operational switches + - `NYASH_USE_NY_COMPILER=1`(selfhost compiler 経路ON) + - `NYASH_JSON_ONLY=1`(子プロセスの余計な出力抑止) + - `NYASH_DISABLE_PLUGINS=1`(必要に応じて子のみ最小化) + - 文分離: 最小ASIルール(深さ0・直前が継続子でない改行のみ終端) + +- Risks / Rollback + - 子プロセス出力がJSONでない→フォールバックで安全運用 + - 代表ケースで parity 不一致→selfhost 経路のみ切替OFF + - 影響範囲: CLI/Runner 層の限定的変更(ゲートOFFなら既存経路と同値) + 【受入(MVP)】 - `tools/ny_roundtrip_smoke.sh` 緑(Case A/B)。 - `apps/tests/esc_dirname_smoke.nyash` / `apps/selfhost/tools/dep_tree_min_string.nyash` を Ny パーサ経路で実行し、PyVM/llvmlite とパリティ一致(stdout/exit)。 +#### 予告: LoopForm(Core‑14)での PHI 自動化(Phase‑15 後) +- LoopForm を強化し、`loop.begin(loop_carried_values) / loop.iter / loop.branch / loop.end` の構造的情報から逆Loweringで PHI を合成。 +- If/短絡についても同様に、構造ブロックから合流点を決めて PHI を自動化。 +- スケジュール: Phase‑15 後(Core‑14)で検討・実装。Phase‑15 では変更しない。 + ### Phase 15.4: VM層のNyash化(PyVMからの置換) - PyVM を足場に、VMコアを Nyash 実装へ段階移植(命令サブセットから) - 動的ディスパッチで13命令処理を目標に拡張 詳細:[セルフホスティング戦略 2025年9月版](implementation/self-hosting-strategy-2025-09.md) +--- + +補足: JSON v0 の扱い(互換) +- Phase‑15: Bridge で PHI を生成(現行継続)。 +- Core‑14 以降: LoopForm で PHI 自動化後、JSON 側の PHI は非必須(将来は除外方向)。 +- 型メタ(“+”の文字列混在/文字列比較)は継続。 + ## 📊 主要成果物 ### コンパイラコンポーネント @@ -65,7 +133,7 @@ MIR 13命令の美しさを最大限に活かし、外部コンパイラ依存 ### 自動生成基盤 - [ ] boxes.yaml(Box型定義) - [ ] externs.yaml(C ABI境界) -- [ ] semantics.yaml(MIR15定義) +- [ ] semantics.yaml(MIR14定義) - [ ] build.rs(自動生成システム) ### ブートストラップ @@ -75,13 +143,21 @@ MIR 13命令の美しさを最大限に活かし、外部コンパイラ依存 ## 🔧 技術的アプローチ -### MIR 13命令の革命 -- **基本演算(5)**: Const, UnaryOp, BinOp, Compare, TypeOp -- **メモリ(2)**: Load, Store -- **制御(4)**: Branch, Jump, Return, Phi -- **Box(1)**: BoxCall(すべての箱操作を統合) -- **外部(1)**: ExternCall - +### MIR 14命令の革命 +1. Const - 定数 + 2. BinOp - 二項演算 + 3. UnaryOp - 単項演算(復活!) + 4. Compare - 比較 + 5. Jump - 無条件ジャンプ + 6. Branch - 条件分岐 + 7. Return - 戻り値 + 8. Phi - SSA合流 + 9. Call - 関数呼び出し + 10. BoxCall - Box操作(配列/フィールド/メソッド統一!) + 11. ExternCall - 外部呼び出し + 12. TypeOp - 型操作 + 13. Safepoint - GC安全点 + 14. Barrier - メモリバリア この究極のシンプルさにより、直接x86変換も現実的に! ### バックエンドの選択肢 @@ -238,6 +314,8 @@ ny_free_buf(buffer) ### ✅ クイックスモーク(現状) - PyVM↔llvmlite パリティ: `tools/parity.sh --lhs pyvm --rhs llvmlite apps/tests/esc_dirname_smoke.nyash` - dep_tree(ハーネスON): `NYASH_LLVM_FEATURE=llvm ./tools/build_llvm.sh apps/selfhost/tools/dep_tree_min_string.nyash -o app_dep && ./app_dep` +- JSON v0 bridge spec: `docs/reference/ir/json_v0.md` +- Stage‑2 smokes: `tools/ny_stage2_bridge_smoke.sh`, `tools/ny_parser_stage2_phi_smoke.sh`, `tools/ny_me_dummy_smoke.sh` ### 📚 関連フェーズ - [Phase 10: Cranelift JIT](../phase-10/) diff --git a/docs/development/roadmap/phases/phase-15/ROADMAP.md b/docs/development/roadmap/phases/phase-15/ROADMAP.md index 0bc0768c..efd7e546 100644 --- a/docs/development/roadmap/phases/phase-15/ROADMAP.md +++ b/docs/development/roadmap/phases/phase-15/ROADMAP.md @@ -35,6 +35,17 @@ This roadmap is a living checklist to advance Phase 15 with small, safe boxes. U - No circular dependency: nyrt provides StringBox/ArrayBox via C ABI - Flag path: `NYASH_USE_NY_COMPILER=1` to switch rust→ny compiler; rust parser as fallback - Add apps/selfhost-compiler/ and minimal smokes + - Stage‑1 checklist: + - [ ] return/int/string/arithmetic/paren JSON v0 emit + - [ ] Minimal ASI(newline separator + continuation tokens) + - [ ] Smokes: `return 1+2*3` / grouping / string literal + - Stage‑2 checklist: + - [ ] local/if/loop/call/method/new/var/logical/compare + - [ ] PHI 合流は Bridge に委譲(If/Loop) + - [ ] Smokes: nested if / loop 累積 / and/or × if/loop +4) PHI 自動化は Phase‑15 後(Core‑14 LoopForm) + - Phase‑15: 現行の Bridge‑PHI を維持し、E2E 緑とパリティを最優先 + - Core‑14: LoopForm 強化+逆Loweringで PHI を自動生成(合流点の定型化) 4) Bootstrap loop (c0→c1→c1') - Use existing trace/hash harness to compare parity; add optional CI gate - **This achieves self-hosting!** Nyash compiles Nyash @@ -63,6 +74,8 @@ This roadmap is a living checklist to advance Phase 15 with small, safe boxes. U - Parser path: `--parser {rust|ny}` or `NYASH_USE_NY_PARSER=1` - JSON dump: `NYASH_DUMP_JSON_IR=1` +- (予告)LoopForm: Core‑14 で仕様化予定 + - Selfhost compiler: `NYASH_USE_NY_COMPILER=1`, child quiet: `NYASH_JSON_ONLY=1` - Load Ny plugins: `NYASH_LOAD_NY_PLUGINS=1` / `--load-ny-plugins` - AOT smoke: `CLIF_SMOKE_RUN=1` @@ -83,6 +96,8 @@ This roadmap is a living checklist to advance Phase 15 with small, safe boxes. U - v0 E2E green (parser pipe + direct bridge) including Ny compiler MVP switch - v1 minimal samples pass via JSON bridge - AOT P2: emit→link→run stable for constant/arith +- Phase‑15 STOP には PHI 切替を含めない(PHI は LoopForm/Core‑14 で扱う) + - 15.3: Stage‑1 代表サンプル緑 + Bootstrap smoke(フォールバック許容)+ 文分離ポリシー公開 - Docs/recipes usable on Windows/Unix ## Notes diff --git a/docs/development/roadmap/phases/phase-15/imports-namespace-plan.md b/docs/development/roadmap/phases/phase-15/imports-namespace-plan.md new file mode 100644 index 00000000..488ca0f1 --- /dev/null +++ b/docs/development/roadmap/phases/phase-15/imports-namespace-plan.md @@ -0,0 +1,38 @@ +# Phase 15.3 — Imports/Namespace/nyash.toml Plan + +Status: 15.3 planning; focus remains on Stage‑1/2 compiler MVP. This document scopes when and how to bring `nyash.toml`/include/import/namespace into the selfhost path without destabilizing parity. + +Goals +- Keep runner‑level `nyash.toml` parsing/resolution as the source of truth during 15.3. +- Accept `using/import` syntax in the Ny compiler as a no‑op (record only) until resolution is delegated. +- Avoid VM changes; resolution happens before codegen. + +Scope & Sequence (Phase 15.3) +1) Stage‑1/2 compiler stability (primary) + - Ny→JSON v0 → Bridge → PyVM/llvmlite parity maintained + - PHI merge remains on Bridge (If/Loop) +2) Imports/Namespace minimal acceptance (15.3‑late) + - Parse `using ns` / `using "path" [as alias]` as statements in the Ny compiler + - Do not resolve; emit no JSON entries (or emit metadata) — runner continues to strip/handle + - Gate via `NYASH_ENABLE_USING=1` +3) Runner remains in charge + - Keep existing Rust runner’s `using` stripping + modules registry population + - `nyash.toml` parsing stays in Rust (Phase 15) + +Out of scope (Phase 15) +- Porting `nyash.toml` parsing to Ny +- Cross‑module codegen/linking in Ny compiler +- Advanced include resolution / package graph + +Acceptance (15.3) +- Ny compiler can lex/parse `using` forms without breaking Stage‑1/2 programs +- Runner path (Rust) continues to resolve `using` and `nyash.toml` as before (parity unchanged) + +Looking ahead (Core‑14 / Phase 16) +- Evaluate moving `nyash.toml` parsing to Ny as a library box (ConfigBox) +- Unify include/import/namespace into a single resolver pass in Ny with a small JSON side channel back to the runner +- Keep VM unchanged; all resolution before MIR build + +Switches +- `NYASH_ENABLE_USING=1` — enable `using` acceptance in Ny compiler (no resolution) +- `NYASH_SKIP_TOML_ENV=1` — skip applying [env] in nyash.toml (existing) diff --git a/docs/guides/language-guide.md b/docs/guides/language-guide.md index 2f525d5c..5bb8a17c 100644 --- a/docs/guides/language-guide.md +++ b/docs/guides/language-guide.md @@ -52,3 +52,7 @@ When you need the implementation details - Tokenizer: src/tokenizer.rs - Parser: src/parser/expressions.rs, src/parser/statements.rs - Lowering to MIR: src/mir/builder/** +Statement Separation (Semicolons) +- Newline separates statements by default; semicolons are optional. +- Use semicolons only when placing multiple statements on one line. +- Minimal ASI rules: newline does not end a statement when the line ends with an operator/dot/comma, or while inside grouping. diff --git a/docs/phase-15/cranelift/CRANELIFT_TASKS.md b/docs/phase-15/cranelift/CRANELIFT_TASKS.md deleted file mode 100644 index f5f37cfe..00000000 --- a/docs/phase-15/cranelift/CRANELIFT_TASKS.md +++ /dev/null @@ -1,25 +0,0 @@ -# Cranelift / AOT/JIT‑AOT Tasks (Phase 15) - -このドキュメントは Cranelift backend(AOT/JIT‑AOT)関連の課題・進捗を集約します。 -selfhosting‑dev ブランチでは VM/JIT 中心で開発するため、詳細はこちらへ集約し、`CURRENT_TASK.md` は軽量化しました。 - -最終更新: 2025‑09‑06(CURRENT_TASK から分離) - -参考リンク -- 旧コンテンツ・完全版アーカイブ: `../../archives/CURRENT_TASK-2025-09-06.md` -- フェーズ概要: `../README.md` - -現状サマリ(抜粋) -- StringBox.length/len が 0 になるケースの是正(Lower 二段フォールバック: string.len_h → any.length_h) -- Hostcall registry/extern thunks の追補(`SYM_STRING_LEN_H` 登録) -- AOT でのまれな segfault(DT_TEXTREL 警告)の追跡(TLS/extern 紐付け順) - -優先課題(案) -1) Return 材化の強化(JIT‑direct/JIT‑AOT 共通) -2) Cranelift import シンボル解決の検証(`extern_thunks::nyash_string_len_h` の実呼出し保証) -3) AOT ツールチェーン(リンク・フラグ)の最小安定セット定義 - -運用メモ -- selfhosting‑dev では本ファイルの参照のみ(直接の実装変更は Cranelift 専用ブランチで実施)。 -- 共有面(ランナー/IR など)に変更が必要な場合は feature gate と互換 API を優先し、両ブランチが同時に衝突しない形へ調整。 - diff --git a/docs/reference/ir/json_v0.md b/docs/reference/ir/json_v0.md new file mode 100644 index 00000000..18c39dae --- /dev/null +++ b/docs/reference/ir/json_v0.md @@ -0,0 +1,81 @@ +# Ny JSON IR v0 — Minimal Spec (Stage‑2) + +Status: experimental but stable for Phase‑15 Stage‑2. Input to `--ny-parser-pipe`. + +Version and root +- `version`: 0 +- `kind`: "Program" +- `body`: array of statements + +Statements (`StmtV0`) +- `Return { expr }` +- `Extern { iface, method, args[] }` (optional; passes through to `ExternCall`) +- `Expr { expr }` (expression statement; side effects only) +- `Local { name, expr }` (Stage‑2) +- `If { cond, then: Stmt[], else?: Stmt[] }` (Stage‑2) +- `Loop { cond, body: Stmt[] }` (Stage‑2; while(cond) body) + +Expressions (`ExprV0`) +- `Int { value }` where `value` is JSON number or digit string +- `Str { value: string }` +- `Bool { value: bool }` +- `Binary { op: "+"|"-"|"*"|"/", lhs, rhs }` +- `Compare { op: "=="|"!="|"<"|"<="|">"|">=", lhs, rhs }` +- `Logical { op: "&&"|"||", lhs, rhs }` (short‑circuit) +- `Call { name: string, args[] }` (function by name) +- `Method { recv: Expr, method: string, args[] }` (box method) +- `New { class: string, args[] }` (construct Box) +- `Var { name: string }` + +CFG conventions (lowered by the bridge) +- If: create `then_bb`, `else_bb`, `merge_bb`. Both branches jump to merge if unterminated. +- Loop: `preheader -> cond_bb -> (body_bb or exit_bb)`, body jumps back to cond. +- Short‑circuit Logical: create `rhs_bb`, `fall_bb`, `merge_bb` with constants on fall path. +- All blocks end with a terminator (branch/jump/return). + +PHI merging (current behavior) +- If: locals updated in `then`/`else` merge at `merge_bb` via `phi`. + - Else欠落時は else 側に分岐前(base)を採用。 + - 片側にしか存在しない新規変数はスコープ外として外へ未伝播。 +- Loop: `cond_bb` にヘッダ PHI を先置き(preheader/base と latch/body end を合流)。 +- 目的: Stage‑2 を早期に安定化させるための橋渡し。将来(Core‑14)は LoopForm からの逆LoweringでPHI自動化予定。 + +Type meta (emitter/LLVM harness cooperation) +- `+` with any string operand → string concat path(handle固定)。 +- `==/!=` with both strings → string compare path。 + +Special notes +- `Var("me")`: Bridge 既定では未定義エラー。デバッグ用に `NYASH_BRIDGE_ME_DUMMY=1` でダミー `NewBox{class}` を注入可(`NYASH_BRIDGE_ME_CLASS` 省略時は `Main`)。 +- `--ny-parser-pipe` は stdin の JSON v0 を受け取り、MIR→MIR‑Interp 経由で実行する。 + +CLI/Env cheatsheet +- Pipe: `echo '{...}' | target/release/nyash --ny-parser-pipe` +- File: `target/release/nyash --json-file sample.json` +- Verbose MIR dump: `NYASH_CLI_VERBOSE=1` +- me dummy: `NYASH_BRIDGE_ME_DUMMY=1 NYASH_BRIDGE_ME_CLASS=ConsoleBox` + +Examples + +Arithmetic +```json +{"version":0,"kind":"Program","body":[ + {"type":"Return","expr":{ + "type":"Binary","op":"+", + "lhs":{"type":"Int","value":1}, + "rhs":{"type":"Binary","op":"*","lhs":{"type":"Int","value":2},"rhs":{"type":"Int","value":3}} + }} +]} +``` + +If with local + PHI merge +```json +{"version":0,"kind":"Program","body":[ + {"type":"Local","name":"x","expr":{"type":"Int","value":1}}, + {"type":"If","cond":{"type":"Compare","op":"<","lhs":{"type":"Int","value":1},"rhs":{"type":"Int","value":2}}, + "then":[{"type":"Local","name":"x","expr":{"type":"Int","value":10}}], + "else":[{"type":"Local","name":"x","expr":{"type":"Int","value":20}}] + }, + {"type":"Return","expr":{"type":"Var","name":"x"}} +]} +``` + diff --git a/docs/reference/language/README.md b/docs/reference/language/README.md index 2f584bc8..cc07d7f4 100644 --- a/docs/reference/language/README.md +++ b/docs/reference/language/README.md @@ -10,6 +10,9 @@ This is the entry point for Nyash language documentation. - Sugar Transformations (?., ??, |> and friends): parser/sugar.rs (source) and tools/nyfmt/NYFMT_POC_ROADMAP.md - Peek Expression Design/Usage: covered in the Language Reference and Phase 12.7 specs above +Statement separation and semicolons +- See: reference/language/statements.md — newline as primary separator; semicolons optional for multiple statements on one line; minimal ASI rules. + Related implementation notes - Tokenizer: src/tokenizer.rs - Parser (expressions/statements): src/parser/expressions.rs, src/parser/statements.rs diff --git a/docs/reference/language/statements.md b/docs/reference/language/statements.md new file mode 100644 index 00000000..50ff878a --- /dev/null +++ b/docs/reference/language/statements.md @@ -0,0 +1,70 @@ +# Statement Separation and Semicolons + +Status: Adopted for Phase 15.3+; parser implementation is staged. + +Policy +- Newline as primary statement separator. +- Semicolons are optional and only needed when multiple statements appear on one physical line. +- Minimal ASI (auto semicolon insertion) rules to avoid surprises. + +Rules (minimal and predictable) +- Newline ends a statement when: + - Parenthesis/brace/bracket depth is 0, and + - The line does not end with a continuation token (`+ - * / . ,` etc.). +- Newline does NOT end a statement when: + - Inside any open grouping `(...)`, `[...]`, `{...}`; or + - The previous token is a continuation token. +- `return/break/continue` end the statement at newline unless the value is on the same line or grouped via parentheses. +- `if/else` (and similar paired constructs): do not insert a semicolon between a block and a following `else`. +- One‑line multi‑statements are allowed with semicolons: `x = 1; y = 2; print(y)`. +- Method chains can break across lines after a dot: `obj\n .method()` (newline treated as whitespace). + +Style guidance +- Prefer newline separation (no semicolons) for readability. +- Use semicolons only when placing multiple statements on a single line. + +Examples +```nyash +// Preferred (no semicolons) +local x = 5 +x = x + 1 +print(x) + +// One line with multiple statements (use semicolons) +local x = 5; x = x + 1; print(x) + +// Line continuation by operator +local n = 1 + + 2 + + 3 + +// Grouping across lines +return ( + 1 + 2 + 3 +) + +// if / else on separate lines without inserting a semicolon +if cond { + x = x - 1 +} +else { + print(x) +} + +// Dot chain across lines +local v = obj + .methodA() + .methodB(42) +``` + +Implementation notes (parser) +- Tokenizer keeps track of grouping depth. +- At newline, attempt ASI only when depth==0 and previous token is not a continuation. +- Error messages should suggest adding a continuation token or grouping when a newline unexpectedly ends a statement. + +Parser dev notes (Stage‑1/2) +- return + newline: treat bare `return` as statement end. To return an expression on the next line, require grouping with parentheses. +- if/else: never insert a semicolon between a closed block and `else` (ASI禁止箇所)。 +- Dot chains: treat `.` followed by newline as whitespace (line continuation)。 +- One‑line multi‑statements: accept `;` as statement separator, but formatter should prefer newlines. +- Unary minus: disambiguate from binary minus; implement after Stage‑1(当面は括弧で回避)。 diff --git a/src/runner/json_v0_bridge.rs b/src/runner/json_v0_bridge.rs index 0fea754e..2cd46dfc 100644 --- a/src/runner/json_v0_bridge.rs +++ b/src/runner/json_v0_bridge.rs @@ -242,10 +242,23 @@ fn lower_expr_with_vars( match e { ExprV0::Var { name } => { if let Some(&vid) = vars.get(name) { - Ok((vid, cur_bb)) - } else { - Err(format!("undefined variable: {}", name)) + return Ok((vid, cur_bb)); } + if name == "me" { + // Optional gate: allow a dummy 'me' instance for Stage-2 JSON smoke + if std::env::var("NYASH_BRIDGE_ME_DUMMY").ok().as_deref() == Some("1") { + let class = std::env::var("NYASH_BRIDGE_ME_CLASS").unwrap_or_else(|_| "Main".to_string()); + let dst = f.next_value_id(); + if let Some(bb) = f.get_block_mut(cur_bb) { + bb.add_instruction(MirInstruction::NewBox { dst, box_type: class, args: vec![] }); + } + vars.insert("me".to_string(), dst); + return Ok((dst, cur_bb)); + } else { + return Err("undefined 'me' outside box context (set NYASH_BRIDGE_ME_DUMMY=1 to inject placeholder)".into()); + } + } + Err(format!("undefined variable: {}", name)) } ExprV0::Call { name, args } => { // Lower args diff --git a/src/runner/modes/common.rs b/src/runner/modes/common.rs index 61d83be3..53720485 100644 --- a/src/runner/modes/common.rs +++ b/src/runner/modes/common.rs @@ -36,6 +36,14 @@ fn suggest_in_base(base: &str, leaf: &str, out: &mut Vec) { impl NyashRunner { /// File-mode dispatcher (thin wrapper around backend/mode selection) pub(crate) fn run_file(&self, filename: &str) { + // Phase-15.3: Ny compiler MVP (Ny -> JSON v0) behind env gate + if std::env::var("NYASH_USE_NY_COMPILER").ok().as_deref() == Some("1") { + if self.try_run_ny_compiler_pipeline(filename) { + return; + } else if std::env::var("NYASH_CLI_VERBOSE").ok().as_deref() == Some("1") { + eprintln!("[ny-compiler] fallback to default path (MVP unavailable for this input)"); + } + } // Direct v0 bridge when requested via CLI/env let use_ny_parser = self.config.parser_ny || std::env::var("NYASH_USE_NY_PARSER").ok().as_deref() == Some("1"); if use_ny_parser { @@ -141,6 +149,84 @@ impl NyashRunner { } } + /// Phase-15.3: Attempt Ny compiler pipeline (Ny -> JSON v0 via Ny program), then execute MIR + fn try_run_ny_compiler_pipeline(&self, filename: &str) -> bool { + use std::io::Write; + // Read input source + let code = match fs::read_to_string(filename) { + Ok(c) => c, + Err(e) => { eprintln!("[ny-compiler] read error: {}", e); return false; } + }; + // Write to tmp/ny_parser_input.ny (as expected by Ny parser v0) + let tmp_dir = std::path::Path::new("tmp"); + if let Err(e) = std::fs::create_dir_all(tmp_dir) { + eprintln!("[ny-compiler] mkdir tmp failed: {}", e); + return false; + } + let tmp_path = tmp_dir.join("ny_parser_input.ny"); + match std::fs::File::create(&tmp_path) { + Ok(mut f) => { + if let Err(e) = f.write_all(code.as_bytes()) { + eprintln!("[ny-compiler] write tmp failed: {}", e); + return false; + } + } + Err(e) => { eprintln!("[ny-compiler] open tmp failed: {}", e); return false; } + } + // Locate current exe to invoke Ny VM for the Ny parser program + let exe = match std::env::current_exe() { + Ok(p) => p, + Err(e) => { eprintln!("[ny-compiler] current_exe failed: {}", e); return false; } + }; + // Prefer new selfhost-compiler entry; fallback to legacy ny_parser_v0 + let cand_new = std::path::Path::new("apps/selfhost-compiler/compiler.nyash"); + let cand_old = std::path::Path::new("apps/selfhost/parser/ny_parser_v0/main.nyash"); + let parser_prog = if cand_new.exists() { cand_new } else { cand_old }; + if !parser_prog.exists() { eprintln!("[ny-compiler] compiler program not found: {}", parser_prog.display()); return false; } + let mut cmd = std::process::Command::new(exe); + cmd.arg("--backend").arg("vm").arg(parser_prog); + // Propagate minimal env; disable plugins to reduce noise + cmd.env_remove("NYASH_USE_NY_COMPILER"); + // Suppress parent runner's result printing in child + cmd.env("NYASH_JSON_ONLY", "1"); + let out = match cmd.output() { + Ok(o) => o, + Err(e) => { eprintln!("[ny-compiler] spawn failed: {}", e); return false; } + }; + if !out.status.success() { + if let Ok(s) = String::from_utf8(out.stderr) { eprintln!("[ny-compiler] parser stderr:\n{}", s); } + return false; + } + let stdout = match String::from_utf8(out.stdout) { Ok(s) => s, Err(_) => String::new() }; + let mut json_line = String::new(); + for line in stdout.lines() { + let t = line.trim(); + if t.starts_with('{') && t.contains("\"version\":0") { json_line = t.to_string(); break; } + } + if json_line.is_empty() { + if std::env::var("NYASH_CLI_VERBOSE").ok().as_deref() == Some("1") { + let head: String = stdout.chars().take(200).collect(); + eprintln!("[ny-compiler] JSON not found in child stdout (head): {}", head.replace('\n', "\\n")); + } + return false; + } + // Parse JSON v0 → MIR module + match json_v0_bridge::parse_json_v0_to_module(&json_line) { + Ok(module) => { + if std::env::var("NYASH_CLI_VERBOSE").ok().as_deref() == Some("1") { + println!("🚀 Ny compiler MVP (ny→json_v0) path ON"); + } + json_v0_bridge::maybe_dump_mir(&module); + self.execute_mir_module(&module); + true + } + Err(e) => { + eprintln!("[ny-compiler] JSON parse failed: {}", e); + false + } + } + } + /// Execute Nyash file with interpreter (common helper) pub(crate) fn execute_nyash_file(&self, filename: &str) { let quiet_pipe = std::env::var("NYASH_JSON_ONLY").ok().as_deref() == Some("1"); diff --git a/tools/ny_me_dummy_smoke.sh b/tools/ny_me_dummy_smoke.sh new file mode 100644 index 00000000..425d6947 --- /dev/null +++ b/tools/ny_me_dummy_smoke.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +ROOT_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd) +BIN="$ROOT_DIR/target/release/nyash" + +if [[ ! -x "$BIN" ]]; then + echo "[build] nyash (release) ..." >&2 + cargo build --release >/dev/null +fi + +TMP_DIR="$ROOT_DIR/tmp" +mkdir -p "$TMP_DIR" + +# Enable me dummy injection for entire run +export NYASH_BRIDGE_ME_DUMMY=1 +export NYASH_BRIDGE_ME_CLASS=ConsoleBox + +# Case 1: me bound to a var and unused (ensures Var("me") resolves) +cat >"$TMP_DIR/me_dummy_bind_only.ny" <<'NY' +local x = me +return 0 +NY + +OUT1=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/me_dummy_bind_only.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT1" | rg -q '^Result:\s*0\b' && echo "✅ me dummy (bind only) OK" || { echo "❌ me dummy (bind only) FAILED"; echo "$OUT1"; exit 1; } + +# Case 2: me used inside an if branch +cat >"$TMP_DIR/me_dummy_in_if.ny" <<'NY' +if 1 < 2 { + local y = me +} +return 0 +NY + +OUT2=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/me_dummy_in_if.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT2" | rg -q '^Result:\s*0\b' && echo "✅ me dummy (in if) OK" || { echo "❌ me dummy (in if) FAILED"; echo "$OUT2"; exit 1; } + +echo "All me-dummy smokes PASS" >&2 diff --git a/tools/ny_parser_stage2_phi_smoke.sh b/tools/ny_parser_stage2_phi_smoke.sh index 38c93ad2..717624b7 100644 --- a/tools/ny_parser_stage2_phi_smoke.sh +++ b/tools/ny_parser_stage2_phi_smoke.sh @@ -41,5 +41,34 @@ NY OUT2=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/phi_loop_sample.ny" | "$BIN" --ny-parser-pipe || true) echo "$OUT2" | rg -q '^Result:\s*3\b' && echo "✅ Loop PHI merge OK" || { echo "❌ Loop PHI merge FAILED"; echo "$OUT2"; exit 1; } -echo "All Stage-2 PHI smokes PASS" >&2 +# If (no-else) PHI merge with base +cat >"$TMP_DIR/phi_if_noelse_sample.ny" <<'NY' +local x = 1 +if 1 > 2 { + local x = 10 +} +return x +NY +OUT3=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/phi_if_noelse_sample.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT3" | rg -q '^Result:\s*1\b' && echo "✅ If(no-else) PHI merge OK" || { echo "❌ If(no-else) PHI merge FAILED"; echo "$OUT3"; exit 1; } + +# Nested If PHI merge +cat >"$TMP_DIR/phi_if_nested_sample.ny" <<'NY' +local x = 1 +if 1 < 2 { + if 2 < 1 { + local x = 100 + } else { + local x = 200 + } +} else { + local x = 300 +} +return x +NY + +OUT4=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/phi_if_nested_sample.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT4" | rg -q '^Result:\s*200\b' && echo "✅ Nested If PHI merge OK" || { echo "❌ Nested If PHI merge FAILED"; echo "$OUT4"; exit 1; } + +echo "All Stage-2 PHI smokes PASS" >&2 diff --git a/tools/ny_stage2_bridge_smoke.sh b/tools/ny_stage2_bridge_smoke.sh new file mode 100644 index 00000000..c0e6fa8b --- /dev/null +++ b/tools/ny_stage2_bridge_smoke.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +ROOT_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd) +BIN="$ROOT_DIR/target/release/nyash" + +if [[ ! -x "$BIN" ]]; then + echo "[build] nyash (release) ..." >&2 + (cd "$ROOT_DIR" && cargo build --release >/dev/null) +fi + +TMP_DIR="$ROOT_DIR/tmp" +mkdir -p "$TMP_DIR" + +pass_case() { echo "✅ $1" >&2; } +fail_case() { echo "❌ $1" >&2; echo "$2" >&2; exit 1; } + +# Case A: arithmetic +printf 'return 1+2*3\n' > "$TMP_DIR/s2_a_arith.ny" +OUT=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/s2_a_arith.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT" | rg -q '^Result:\s*7\b' && pass_case "Stage2 arithmetic" || fail_case "Stage2 arithmetic" "$OUT" + +# Case B: logical and (short-circuit) +cat > "$TMP_DIR/s2_b_and.ny" <<'NY' +return (1 < 2) && (2 < 3) +NY +OUT=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/s2_b_and.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT" | rg -q '^Result:\s*true\b' && pass_case "Stage2 logical AND" || fail_case "Stage2 logical AND" "$OUT" + +# Case C: logical or (short-circuit) +cat > "$TMP_DIR/s2_c_or.ny" <<'NY' +return (1 > 2) || (2 < 3) +NY +OUT=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/s2_c_or.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT" | rg -q '^Result:\s*true\b' && pass_case "Stage2 logical OR" || fail_case "Stage2 logical OR" "$OUT" + +# Case D: compare eq +cat > "$TMP_DIR/s2_d_eq.ny" <<'NY' +return (1 + 1) == 2 +NY +OUT=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/s2_d_eq.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT" | rg -q '^Result:\s*true\b' && pass_case "Stage2 compare ==" || fail_case "Stage2 compare ==" "$OUT" + +# Case E: nested if with locals -> 200 +cat > "$TMP_DIR/s2_e_nested_if.ny" <<'NY' +local x = 1 +if 1 < 2 { + if 2 < 1 { + local x = 100 + } else { + local x = 200 + } +} else { + local x = 300 +} +return x +NY +OUT=$(python3 "$ROOT_DIR/tools/ny_parser_mvp.py" "$TMP_DIR/s2_e_nested_if.ny" | "$BIN" --ny-parser-pipe || true) +echo "$OUT" | rg -q '^Result:\s*200\b' && pass_case "Stage2 nested if" || fail_case "Stage2 nested if" "$OUT" + +echo "All Stage-2 bridge smokes PASS" >&2 + diff --git a/tools/selfhost_compiler_smoke.sh b/tools/selfhost_compiler_smoke.sh new file mode 100644 index 00000000..6c0aef83 --- /dev/null +++ b/tools/selfhost_compiler_smoke.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +ROOT_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd) +BIN="$ROOT_DIR/target/release/nyash" + +if [[ ! -x "$BIN" ]]; then + echo "[build] nyash (release) ..." >&2 + (cd "$ROOT_DIR" && cargo build --release >/dev/null) +fi + +TMP_DIR="$ROOT_DIR/tmp" +mkdir -p "$TMP_DIR" + +printf 'return 1+2*3\n' > "$TMP_DIR/ny_parser_input.ny" + +OUT=$(NYASH_USE_NY_COMPILER=1 NYASH_CLI_VERBOSE=1 "$BIN" --backend vm "$ROOT_DIR/apps/examples/string_p0.nyash" || true) +if echo "$OUT" | rg -q 'ny compiler MVP \(ny→json_v0\) path ON' && echo "$OUT" | rg -q '^Result:\s*0\b'; then + echo "✅ ny compiler MVP path OK (json_v0 emit + result 0)" +else + echo "WARN: ny compiler path not used; fallback executed (acceptable during MVP)" + echo "$OUT" | sed -n '1,120p' +fi + +echo "All selfhost compiler smokes PASS" >&2