Commit Graph

178 Commits

Author SHA1 Message Date
37f93d5630 golden: add for/foreach normalized AST and loop_nonreorder (skip-reorder); docs: MacroCtx section; demo macro (identity) 2025-09-20 09:19:28 +09:00
497da75f90 macroctx: pass ctx JSON (caps) to user macros; add demo macro; docs: MacroCtx section and sandbox Box API; golden for for/foreach normalized AST 2025-09-20 09:11:52 +09:00
8a84339ac2 core: for/foreach -> Loop normalization (always-on); LoopForm MVP-3 per-segment reorder; smokes stabilized (VM + LLVM PHI); docs updated (macro-system, loopform); quiet macro load logs 2025-09-20 08:39:40 +09:00
3ee54470a5 macro(match): add golden for guard literal OR; add smoke; update CURRENT_TASK milestones 2025-09-20 05:36:25 +09:00
c536b402a2 macro(match): add guard/type normalization smoke; update CURRENT_TASK progress 2025-09-20 05:01:58 +09:00
166c374eec macro(if/match): stabilize PeekExpr→If normalization via internal-child; default runner off; propagate child stderr; fix JsonBuilder local_decl; add scope-hints doc; extend PHI smoke; golden for match literal passes 2025-09-20 05:00:31 +09:00
8cb93b9f1f tests(macro): inline samples into new directory hierarchy and drop legacy macro_golden_* sources 2025-09-20 03:37:20 +09:00
3c1486e411 tests(macro): organize under apps/tests/macro/{if,loopform,collections,types,strings,identity,test_runner} with thin include wrappers; update all golden/smoke scripts and docs to new paths 2025-09-20 02:20:02 +09:00
c5ab95d082 docs(cf/pattern): add ControlFlowBuilder and PatternBuilder guides; reference from If/Match normalization; plan updated in CURRENT_TASK 2025-09-20 01:15:26 +09:00
02c8244964 scopebox: add design guide; scaffold no-op scope/defer macro; add MIR hint plan to CURRENT_TASK 2025-09-20 00:59:50 +09:00
1d965c7241 docs(cf-normalize): add If/Match normalization guide and note in CURRENT_TASK Next; confirm pipeline order before LoopForm 2025-09-20 00:36:25 +09:00
7a57abfb95 docs(loopform): add carrier normalization examples and local run tips; task: Next short + 48h Action Items updated for LoopForm MVP-2/3 and LLVM PHI smoke 2025-09-20 00:32:29 +09:00
da32455afc freeze: macro platform complete; default ON with profiles; env consolidation; docs + smokes\n\n- Profiles: --profile {lite|dev|ci|strict} (dev-like default for macros)\n- Macro paths: prefer NYASH_MACRO_PATHS (legacy envs deprecated with warnings)\n- Selfhost pre-expand: auto mode, PyVM-only, add smokes (array/map)\n- Docs: user-macros updated; new macro-profiles guide; AGENTS freeze note; CURRENT_TASK freeze\n- Compat: non-breaking; legacy envs print deprecation notices\n 2025-09-19 22:27:59 +09:00
811e3eb3f8 feat: comprehensive development progress
- Pattern matching implementation extended in match_expr.rs
- CLI configuration structured with categorized groups (task recommendation completed)
- Python LLVM builder split into function_lower.py (task recommendation completed)
- parse_box_declaration massive function refactored (task recommendation completed)
- Phase 16 Macro Revolution comprehensive planning and documentation
- Archive legacy phase documentation for clean structure
- HTTP message box improvements and performance optimizations
- MIR builder enhancements and control flow improvements

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-19 15:11:57 +09:00
3c7a5de900 runner(cli): adopt CliConfig::as_groups across runner modules (dispatch/common/pipe_io/mir/bench). llvm-builder: extract ny_main wrapper to builders.entry; add optional env-gated function_lower delegation; keep default behavior unchanged 2025-09-19 14:29:02 +09:00
5bb7f67c01 perf(typebox): reduce clones with Arc<str> for MethodSignature.name and TypeBox.methods keys; API kept stable. tools: add all-cases empty-PHI smoke runner. parser(box): extract birth_once cycle validation into helper 2025-09-19 12:13:43 +09:00
ef1f5206da tools: add LLVM IR empty-PHI smoke checker; ensures every 'phi i64' has incoming list; used with NYASH_LLVM_DUMP_IR 2025-09-19 12:01:13 +09:00
4f82edf901 llvm: gate empty PHI sanitize behind NYASH_LLVM_SANITIZE_EMPTY_PHI=1; docs: AGENTS.md note. perf: begin clone() reduction plan — TypeBox.name/type_parameters to Arc<str>, adjusted builders/registry; no behavior change 2025-09-19 11:54:28 +09:00
f64718d22a parser(box): extract block-first unified members into helper; preserve cache/poison and birth_once guard; integrate in parse_box_declaration; update CURRENT_TASK.md 2025-09-19 11:08:14 +09:00
7dfd55bfdb pyvm: implement TypeOp(check) + strict match-guard smokes; parser: guard support in match; llvm: PHI wiring at block head + incoming normalization; docs: AGENTS LLVM/PHI + guard policy; add tests; plan: refactor parse_box_declaration + TODO triage + clone reduction + CLI split + LLVM builder split; update CURRENT_TASK.md 2025-09-19 10:52:57 +09:00
45e1d57536 docs(lang): adopt match as unified pattern matching; remove peek from docs; update EBNF and guides
- Decision recorded in CURRENT_TASK: replace  with  (no compat alias)
- EBNF: add match_expr/pattern/guard; note legacy peek removal
- Guides/Reference: update examples and wording (null-coalesce/safe-access sugar via match)
2025-09-19 07:42:38 +09:00
5e818eeb7e stage3: unify to cleanup; MIR return-defer; docs+smokes updated; LLVM(harness): finalize_phis ownership, ret.py simplified, uses-predeclare; cleanup return override green; method-postfix cleanup return WIP (PHI head) 2025-09-19 02:07:38 +09:00
951a050592 selfhost: introduce using-based imports for compiler/parser/tools; keep includes temporarily. llvm: add PHI wiring JSON trace + unit/integration tests; fast test suite extended. runner: split selfhost helpers, small cleanups. 2025-09-18 13:35:38 +09:00
3fe908eb0d docs/runtime: externcall + println normalization; NyRT silent result flag; PyVM console warn/error/trace routing\nselfhost(M2): add MirEmitterBox (Return(Int)->const+ret) and --emit-mir switch; initial selfhost smoke\nrunner: child args/env hardening for selfhost (NYASH_NY_COMPILER_CHILD_ARGS under --; NYASH_JSON_ONLY=1, NYASH_VM_USE_PY=1, NYASH_DISABLE_PLUGINS=1); VM path strips using lines when NYASH_ENABLE_USING=1\nmeta: clarify PyVM/llvmlite as primary (vm/interpreter legacy OFF) in CURRENT_TASK.md; README links to externcall docs 2025-09-18 06:35:49 +09:00
5d51086530 python-plugin: RAII (PyOwned/PyBorrowed) + autodecode enum; crate: ny-llvmc --emit exe with NyRT link; tools: build_llvm.sh crate-exe path + crate_exe_smoke; CURRENT_TASK update 2025-09-18 03:57:25 +09:00
8aa01668ff feat(plugins): migrate Python family to TypeBox v2; complete first‑party v2 set (Regex/Net/Path/Math/Time/File)\n\nfeat(smoke): add Net round‑trip sample and wire to plugin v2 smoke\n\nfeat(cli): add --emit-mir-json and runner emit+exit path\n\nchore(config): register Python v2 boxes in nyash.toml\n\nchore(loader): improve v2 diagnostics and ensure v2 shim dispatch\n\nchore(build): integrate ny-llvmc env gate in build_llvm.sh\n\nchore(tasks): update CURRENT_TASK with v2 Python/Net/emit flag 2025-09-17 22:01:29 +09:00
4ea3ca2685 docs/ci: selfhost bootstrap/exe-first workflows; add ny-llvmc scaffolding + JSON v0 schema validation; plan: unify to Nyash ABI v2 (no backwards compat) 2025-09-17 20:33:19 +09:00
a5054a271b llvm(py): introduce BuildCtx + trace hub; refactor if-merge prepass + PHI wiring into module; unify logs; ctx-enable compare/ret/call/boxcall/externcall/typeop/newbox/safepoint; curated smoke option for if-merge; README updates; keep behavior stable 2025-09-17 16:11:01 +09:00
2720884a20 bridge/json_v0: split expr lowering; add Ternary/Peek lowering + AST; selfhost Peek JSON emit; add selfhost Peek smoke; warning cleanup in lowering/optimizer/verification
- Split expr lowering into ; route calls from stmt lowering
- Implement ternary/peek lowering (MIR13 PHI-off=Copy, PHI-on=Phi)
- Extend JSON v0 AST (ExprV0::{Ternary,Peek}, PeekArmV0)
- Selfhost parser_box: emit Peek JSON; add Stage-2 'Peek basic' smoke
- Reduce warnings: remove unused imports/vars in several modules
- current_task: update plan for legacy VM/Interpreter offboarding
2025-09-17 11:45:57 +09:00
d99b941218 docs: add MIR13 mode doc and set PHI-off as default; bridge lowering split (if/loop/try); llvmlite resolver stabilization; curated runner default PHI-off; refresh CURRENT_TASK.md 2025-09-17 10:58:12 +09:00
adbb0201a9 chore(fmt): add legacy stubs and strip trailing whitespace to unblock cargo fmt 2025-09-17 07:43:07 +09:00
f80f6b7796 docs: update CURRENT_TASK.md with logging helpers and next steps (runner trace + selfhost helpers) 2025-09-17 07:12:15 +09:00
c553f2952d runner/env: centralize CLI/env getters; parser expr split (call/primary); verifier utils direct; optimizer: boxfield peephole; LLVM: branch cond normalize hook; add trace macro scaffolding; refactor common.rs verbose checks 2025-09-17 06:55:39 +09:00
9dc5c9afb9 refactor: split optimizer/verifier/parser modules (mainline); add runner trace/directives; add LLVM terminator/select scaffolds; extract AST Span; update CURRENT_TASK with remaining plan 2025-09-17 05:56:33 +09:00
4bd49884ca llvm/codegen: extract wrapper/object emission into object.rs; dedupe mod.rs. runner/json_v0_bridge: introduce BridgeEnv + VarScope to unify lowering paths (lower_expr/args) and cache env flags; thread env through stmt lowering; minor HashMap type cleanups. Build + PyVM stage-2 smokes passed. 2025-09-17 02:50:39 +09:00
adbfbb2c76 ci: add GitHub Actions workflow; runner: prefer PyVM in selfhost paths; stage3: add LLVM + bridge acceptance smokes; docs: update env flags 2025-09-17 01:20:15 +09:00
5c9213cd03 smokes: add curated LLVM runner; archive legacy smokes; PHI-off unified across Bridge/Builder; LLVM resolver tracing; minimal Throw lowering; config env getters; dev profile and root cleaner; docs updated; CI workflow runs curated LLVM (PHI-on/off) 2025-09-16 23:49:36 +09:00
97a76c0571 selfhost: stub stage3 parser nodes under gate 2025-09-16 18:36:54 +09:00
fa1619bf4b selfhost: add ternary parser and plugin prefix guard 2025-09-16 17:38:22 +09:00
fce0d2cdef docs: log parser/emitter Stage-2 update 2025-09-16 17:11:03 +09:00
63c8fda808 🔍 Research: GPT-5-Codex capabilities and GitHub PR integration
## Summary
Investigated OpenAI's new GPT-5-Codex model and Codex GitHub PR review integration capabilities.

## GPT-5-Codex Analysis

### Benchmark Performance (Good)
- SWE-bench Verified: 74.5% (vs GPT-5's 72.8%)
- Refactoring tasks: 51.3% (vs GPT-5's 33.9%)
- Code review: Higher developer ratings

### Real-World Issues (Concerning)
- Users report degraded coding performance
- Scripts that previously worked now fail
- Less consistent than GPT-4.5
- Longer response times (minutes vs instant)
- "Creatively and emotionally flat"
- Basic errors (e.g., counting letters incorrectly)

### Key Finding
Classic case of "optimizing for benchmarks vs real usability" - scores well on tests but performs poorly in practice.

## Codex GitHub PR Integration

### Setup Process
1. Enable MFA and connect GitHub account
2. Authorize Codex GitHub app for repos
3. Enable "Code review" in repository settings

### Usage Methods
- **Manual**: Comment '@codex review' in PR
- **Automatic**: Triggers when PR moves from draft to ready

### Current Limitations
- One-way communication (doesn't respond to review comments)
- Prefers creating new PRs over updating existing ones
- Better for single-pass reviews than iterative feedback

## 'codex resume' Feature
New session management capability:
- Resume previous codex exec sessions
- Useful for continuing long tasks across days
- Maintains context from interrupted work

🐱 The investigation reveals that while GPT-5-Codex shows benchmark improvements, practical developer experience has declined - a reminder that metrics don't always reflect real-world utility\!
2025-09-16 16:28:25 +09:00
47f4ca0e44 📚 docs: Record field declaration design discussion in papers
## Summary
Documented the "init block vs fields-at-top" design discussion as a valuable example of AI-human collaboration in language design.

## Changes

### Paper G (AI Collaboration)
- Added field-declaration-design.md documenting the entire discussion flow
- Showcased how complex init block proposal evolved to simple "fields at top" rule
- Demonstrates AI's tendency toward complexity vs human intuition for simplicity

### Paper H (AI Practical Patterns)
- Added Pattern #17: "Gradual Refinement Pattern" (段階的洗練型)
- Documents the process: Complex AI proposal → Detailed analysis → Human insight → Convergence
- Field declaration design as a typical example

### Paper K (Explosive Incidents)
- Added Incident #046: "init block vs fields-at-top incident"
- Updated total count to 46 incidents
- Shows how a single human comment redirected entire design approach

## Design Decision
After analysis, decided that BoxIndex should remain a compiler-internal structure, not a core Box:
- Core Boxes: User-instantiable runtime values (String, Integer, Array, Map)
- Compiler internals: BoxIndex for name resolution (compile-time only)
- Clear separation of concerns between language features and compiler tools

## Philosophy
This discussion exemplifies key principles:
- The best design needs no explanation
- Constraints provide clarity, not limitation
- "Everything is Box" doesn't mean "compiler internals are Boxes"
- AI tends toward theoretical completeness; humans toward practical simplicity

🐱 Sometimes the simplest answer is right in front of us\!
2025-09-16 14:57:05 +09:00
6ca56b0652 feat: 配列/Mapリテラル糖衣構文の実装とネームスペース解決の改善計画
- ArrayLiteral/MapLiteralのAST定義追加
- パーサーで[...]と{...}構文をサポート
- MIR Builderでnew Box() + push/setへのdesugaring実装
- テストケースとスモークスクリプト追加
- CURRENT_TASK.mdにネームスペース解決Phase-1計画を追記
- 3段階解決順序(ローカル→エイリアス→プラグイン)の設計合意
2025-09-16 06:13:44 +09:00
f9bd13f4e4 docs: Update CURRENT_TASK.md with refactoring progress
- Add module splitting completion status (runner module)
- Record successful build after phi visibility fixes
- Update incident log with 4 new entries (042-045)
- Document PyVM vs EXE-first strategy confusion resolution
- Track Phase 15.3 (Nyash compiler MVP) progress

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-16 02:01:19 +09:00
94d95dfbcd 🚀 Break/Continue/Try-Catch構文のサポート追加とMIRループ制御強化
## 主な変更点

### 🎯 MIRループ制御の実装(根治対応)
- src/mir/loop_builder.rs: Break/Continue対応のループコンテキスト管理
  - ループのbreak/continueターゲットブロック追跡
  - ネストループの適切な処理
- src/mir/builder.rs: Break/Continue文のMIR生成実装
- src/tokenizer.rs: Break/Continue/Tryトークン認識追加

### 📝 セルフホストパーサーの拡張
- apps/selfhost-compiler/boxes/parser_box.nyash:
  - Stage-3: break/continue構文受理(no-op実装)
  - Stage-3: try-catch-finally構文受理(構文解析のみ)
  - エラー処理構文の将来対応準備

### 📚 ドキュメント更新
- 論文K(爆速事件簿): 45事例に更新(4件追加)
  - PyVM迂回路の混乱事件
  - Break/Continue無限ループ事件
  - EXE-first戦略の再発見
- 論文I(開発秘話): Day 38の重要決定追加

### 🧪 テストケース追加
- apps/tests/: ループ制御とPHIのテストケース
  - nested_loop_inner_break_isolated.nyash
  - nested_loop_inner_continue_isolated.nyash
  - loop_phi_one_sided.nyash
  - shortcircuit関連テスト

## 技術的詳細
- Break/ContinueをMIRレベルで適切に処理
- 無限ループ問題(CPU 99.9%暴走)の根本解決
- 将来の例外処理機能への準備

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-15 22:14:42 +09:00
d90216e9c4 📚 Phase 15 - セルフホスティング戦略の明確化とEXE-first実装
## 主な変更点

### 🎯 戦略の転換と明確化
- PyVMを開発ツールとして位置づけ(本番経路ではない)
- EXE-first戦略を明確に優先(build_compiler_exe.sh実装済み)
- Phase順序の整理: 15.2(LLVM)→15.3(コンパイラ)→15.4(VM)

### 🚀 セルフホスティング基盤の実装
- apps/selfhost-compiler/にNyashコンパイラMVP実装
  - compiler.nyash: メインエントリー(位置引数対応)
  - boxes/: parser_box, emitter_box, debug_box分離
- tools/build_compiler_exe.sh: ネイティブEXEビルド+dist配布
- Python MVPパーサーStage-2完成(local/if/loop/call/method/new)

### 📝 ドキュメント整備
- Phase 15 README/ROADMAP更新(Self-Hosting優先明記)
- docs/guides/exe-first-wsl.md: WSLクイックスタート追加
- docs/private/papers/: 論文G~L、爆速事件簿41事例収録

### 🔧 技術的改善
- JSON v0 Bridge: If/Loop PHI生成実装(ChatGPT協力)
- PyVM/llvmliteパリティ検証スイート追加
- using/namespace機能(gated実装、Phase 15では非解決)

## 次のステップ
1. パーサー無限ループ修正(未実装関数の実装)
2. EXEビルドとセルフホスティング実証
3. c0→c1→c1'ブートストラップループ確立

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-15 18:44:49 +09:00
af11c6855b 🚀 Start Phase 15.3: Nyash compiler MVP implementation
Major milestone:
- Set up apps/selfhost-compiler/ directory structure
- Implement basic Nyash compiler in Nyash (CompilerBox)
- Stage-1: Basic arithmetic parser (int/string/+/-/*/括弧/return)
- JSON v0 output compatible with --ny-parser-pipe
- Runner integration with NYASH_USE_NY_COMPILER=1 flag
- Comprehensive smoke tests for PHI/Bridge/Stage-2

Technical updates:
- Updated CLAUDE.md with Phase 15.3 status and MIR14 details
- Statement separation policy: newline-based with minimal ASI
- Fixed runaway ny-parser-pipe processes (CPU 94.9%)
- Clarified MIR14 as canonical instruction set (not 13/18)
- LoopForm strategy: PHI auto-generation during reverse lowering

Collaborative development:
- ChatGPT5 implementing compiler skeleton
- Codex provided LoopForm PHI generation guidance
- Claude maintaining documentation and coordination

🎉 セルフホスティングの歴史的一歩!自分自身をコンパイルする日が近いにゃ!

Co-Authored-By: ChatGPT <noreply@openai.com>
2025-09-15 01:21:37 +09:00
d01f9b9c93 Implement JSON v0 Bridge with full PHI support for If/Loop statements
Major implementation by ChatGPT:
- Complete JSON v0 Bridge layer with PHI generation for control flow
- If statement: Merge PHI nodes for variables updated in then/else branches
- Loop statement: Header PHI nodes for loop-carried dependencies
- Python MVP Parser Stage-2: Added local/if/loop/call/method/new support
- Full CFG guarantee: All blocks have proper terminators (branch/jump/return)
- Type metadata for string operations (+, ==, !=)
- Comprehensive PHI smoke tests for nested and edge cases

This allows MIR generation without Rust MIR builder - massive step towards
eliminating Rust build dependency!

🎉 ChatGPTが30分以上かけて実装してくれたにゃ!

Co-Authored-By: ChatGPT <noreply@openai.com>
2025-09-14 23:22:05 +09:00
5cad0ab20c 🔧 Fix peek expression terminator issues and add ternary operator support
- Fix LLVM IR terminator missing in peek expression entry blocks
- Add proper jump instructions between peek blocks
- Implement ternary operator (? :) as syntactic sugar for peek
- Update Python LLVM externcall handling for improved compatibility
- Add comprehensive test cases for peek and ternary expressions
- Update language guide with ternary operator documentation

ChatGPTが頑張って修正してくれたにゃ!🐱

Co-Authored-By: ChatGPT <noreply@openai.com>
2025-09-14 20:30:38 +09:00
8f1b2ffa12 📚 docs: Update CLAUDE.md with latest progress and discoveries
Major updates:
- Updated progress section (2025-09-14) with recent achievements:
  - Python LLVM backend reaching production level (all tests passing)
  - Nyash parser implementation started by ChatGPT5
  - peek expression rediscovery (when→peek rename)
  - Box theory simplifying SSA construction (650→100 lines)
  - AI collaboration paper completed

Documentation improvements:
- Changed 500-line limit to "guideline" (not strict requirement)
- Upgraded Python LLVM backend from "experimental" to "production level"
- Added peek expression examples and usage patterns
- Updated todo list status (parser planning completed, compiler MVP in progress)

These updates reflect the significant progress towards self-hosting,
especially the parser implementation using peek expressions to avoid
19-line if-else nesting.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-14 19:26:46 +09:00