freeze: macro platform complete; default ON with profiles; env consolidation; docs + smokes\n\n- Profiles: --profile {lite|dev|ci|strict} (dev-like default for macros)\n- Macro paths: prefer NYASH_MACRO_PATHS (legacy envs deprecated with warnings)\n- Selfhost pre-expand: auto mode, PyVM-only, add smokes (array/map)\n- Docs: user-macros updated; new macro-profiles guide; AGENTS freeze note; CURRENT_TASK freeze\n- Compat: non-breaking; legacy envs print deprecation notices\n

This commit is contained in:
Selfhosting Dev
2025-09-19 22:27:59 +09:00
parent 811e3eb3f8
commit da32455afc
192 changed files with 6454 additions and 2973 deletions

View File

@ -0,0 +1,52 @@
# MacroBox in Nyash (Design Draft)
Status: Design draft (Phase 16+). This document specifies the Nyash-side API for user-extensible macros that execute during compilation (pre-MIR). Rust prototype exists; this spec focuses on the Nyash interface and constraints.
Philosophy
- Box Independence (loose coupling): macro expands against public interfaces only; avoid coupling to private internals.
- Deterministic, side-effect free: no IO, no randomness, no time/env dependence.
- Safety and staging: runs in a sandbox before MIR generation; never observes runtime state.
API (Nyash)
```nyash
box MacroBoxSpec {
// Required entry. Receives entire AST and returns a transformed AST.
static function expand(ast) { /* pure transform */ }
// Optional metadata.
static function name() { return "MyMacro" }
}
```
Registration
- Declared in `nyash.toml` (planned):
```
[macros]
paths = [
"apps/macros/my_macro.nyash",
"apps/macros/json_lints.nyash"
]
```
- Loading policy (dev-only gate): `NYASH_MACRO_BOX_NY=1` enables loading Nyash MacroBoxes from configured paths.
- Isolation: loaded in a dedicated interpreter with IO disabled; only AST utilities and safe helpers exposed.
- Interim mapping (prototype): `name()` may map to built-in MacroBoxes for effects (e.g., `"UppercasePrintMacro"`). Otherwise, identity transform.
Execution Order
- Built-in (Rust) macro passes
- User MacroBoxes (Nyash) in registration order
- Test harness injection
- Stop on fixed point, or when reaching the pass/cycle guard limits.
Guards
- Max passes: `NYASH_MACRO_MAX_PASSES` (default 32)
- Cycle window: `NYASH_MACRO_CYCLE_WINDOW` (default 8)
Constraints
- Pure transform: no external calls (FFI/hostcall) and no global writes.
- Public-only policy for derives: when generating methods like `equals` or `toString`, operate on public fields only.
- Diagnostics: write via `NYASH_MACRO_TRACE=1` (compiler-owned), not via print in user code.
Future Work
- Packaging: macro crates with versioning; integrity checks.
- Capability tokens: opt-in capabilities for limited read-only context (e.g., box names list).
- Attribute-style hooks: `@derive`, `@rewrite`, `@lint` as MacroBox conventions.

29
docs/guides/macro-box.md Normal file
View File

@ -0,0 +1,29 @@
# MacroBox — User Extensible Macro Units (Experimental)
Status: Prototype, behind env gates (default OFF).
Goals
- Allow users to implement procedural macro-like expansions in pure Nyash (future) and Rust (prototype) with a clean, deterministic interface.
- Preserve “Box independence” (loose coupling). Operate on public interfaces; avoid relying on private internals.
Enabling
- Enable MacroBox execution: `NYASH_MACRO_BOX=1`
- Enable built-in example: `NYASH_MACRO_BOX_EXAMPLE=1`
- Macro engine is ON by default; disable with `NYASH_MACRO_DISABLE=1`.
API (Rust, prototype)
- Trait: `trait MacroBox { fn name(&self) -> &'static str; fn expand(&self, ast: &ASTNode) -> ASTNode }`
- Register: `macro::macro_box::register(&MY_BOX)`
- Execution: Engine applies all registered boxes once per expansion pass (after built-in expansions). Order is registration order.
Constraints (philosophy)
- Deterministic; side-effect free; no IO.
- Respect Box independence: operate on public interfaces only where applicable.
Built-in example
- `UppercasePrintMacro` (guarded by `NYASH_MACRO_BOX_EXAMPLE=1`): transforms `print("UPPER:...")` to uppercase.
Future plan
- MacroBox in Nyash: `box MacroBox { static function expand(ast) -> ast }` with sandboxing and package registration via `nyash.toml`.
- Attribute-style macros: `@derive`-like hooks executed as MacroBoxes.

View File

@ -0,0 +1,39 @@
# Macro Profiles — Simple, Practical Defaults
Profiles simplify CLI/ENV by choosing sensible defaults for macro execution.
Profiles
- lite
- macros: OFF
- strict: OFF
- trace: OFF
- dev (default behavior)
- macros: ON
- strict: ON
- trace: OFF
- ci / strict
- macros: ON
- strict: ON
- trace: OFF
CLI
- `nyash --profile lite file.nyash`
- `nyash --profile dev file.nyash`
- `nyash --profile ci file.nyash`
ENV mapping (non-breaking; can be overridden)
- lite → `NYASH_MACRO_ENABLE=0`, `NYASH_MACRO_STRICT=0`, `NYASH_MACRO_TRACE=0`
- dev → `NYASH_MACRO_ENABLE=1`, `NYASH_MACRO_STRICT=1`, `NYASH_MACRO_TRACE=0`
- ci/strict → `NYASH_MACRO_ENABLE=1`, `NYASH_MACRO_STRICT=1`, `NYASH_MACRO_TRACE=0`
Recommended ENV (minimal)
- `NYASH_MACRO_ENABLE=1`
- `NYASH_MACRO_PATHS=...`
- `NYASH_MACRO_STRICT=1`
- `NYASH_MACRO_TRACE=0|1`
Deprecated ENV (kept for compatibility for now)
- `NYASH_MACRO_BOX_NY=1` + `NYASH_MACRO_BOX_NY_PATHS=...` → use `NYASH_MACRO_PATHS`
- `NYASH_MACRO_BOX_CHILD_RUNNER` → no longer needed
- `NYASH_MACRO_TOPLEVEL_ALLOW` → default OFF; prefer CLI `--macro-top-level-allow` when necessary

101
docs/guides/macro-system.md Normal file
View File

@ -0,0 +1,101 @@
# Macro System (Phase 16) — Quickstart
Status: MVP available behind environment gates (default OFF). This page describes how to enable and use the initial features (derive, test runner, expansion dump) and the developerfacing Pattern/Quote API preview.
## Enabling/disabling macros
- Default: ON
- Disable: `NYASH_MACRO_DISABLE=1` or `NYASH_MACRO_ENABLE=0|false|off`
- Debug trace: `NYASH_MACRO_TRACE=1`
- CLI shortcut for debug: `--expand` (prints pre/post AST, and sets trace)
## Derive (MVP)
- Provided derives: `Equals`, `ToString`
- Control via env:
- `NYASH_MACRO_DERIVE=Equals,ToString` (default when unset)
- `NYASH_MACRO_DERIVE_ALL=1` (force all derives)
- Behavior:
- Injects `equals(__ny_other)` and `toString()` into Box declarations when not already present.
- Public-only: derives operate on public fields only (Philosophy: Box independence / loose coupling).
- No changes to MIR instruction set; expansion happens on AST, then compiled as usual.
Example
```
NYASH_MACRO_ENABLE=1 ./target/release/nyash --backend vm apps/APP/main.nyash
```
## Test runner (MVP)
- CLI: `--run-tests` → enable macro engine and inject a test harness main
- Discovery targets:
- Toplevel functions with names `test_*`
- Box static/instance methods `test_*`
- Filtering: `--test-filter SUBSTR` or `NYASH_TEST_FILTER=SUBSTR`
- Entry policy when a main exists:
- `--test-entry wrap` → rename original `main` to `__ny_orig_main`, run tests then call original
- `--test-entry override` → replace entry with test harness only
- Force apply even if a main exists: `NYASH_TEST_FORCE=1`
- Return policy in wrap mode:
- `--test-return {tests|original}` or `NYASH_TEST_RETURN`
- `tests` (default): harness returns number of failed tests
- `original`: return the original main result (tests still run)
- Parameterized tests (MVP): `NYASH_TEST_ARGS_DEFAULTS=1` injects integer `0` defaults for each parameter (static/instance)
- Exit code: number of failed tests (0 = green)
Examples
```
# run all tests in a file
./target/release/nyash --run-tests apps/tests/my_tests.nyash
# filter + wrap entry + default arg injection
NYASH_MACRO_ENABLE=1 NYASH_TEST_ARGS_DEFAULTS=1 \
./target/release/nyash --run-tests --test-filter http --test-entry wrap apps/tests/my_tests.nyash
```
## Expansion dump
```
./target/release/nyash --expand --dump-ast apps/tests/ternary_basic.nyash
```
Shows pre/post expansion AST (debug only).
## Developer API (preview)
- Pattern/Quote primitives are available to bootstrap macro authorship.
- TemplatePattern:
- Bind placeholders using `$name` inside a template AST.
- OrPattern: alternation of templates.
- Variadic `$...name` supported at any position inside call/array argument lists; binds the variable-length segment to a pseudo list (`ASTNode::Program`).
- Unquote:
- Replace `$name` with the bound AST.
- Splice `$...name` into call/array argument lists.
- Array/Map nodes participate in pattern/unquote (map keys must match literally; values can bind via `$name`).
### JSON test args (advanced)
For `--run-tests`, you can supply per-test arguments and instance construction details via `NYASH_TEST_ARGS_JSON`.
Shapes:
- Simple list (as before): `{ "test_name": [1, "s", true], "Box.method": [ 0, 1 ] }`
- Detailed object:
- `{ "Box.method": { "args": [ ... ], "instance": { "ctor": "new|birth", "args": [ ... ], "type_args": ["T", "U"] } } }`
- Typed values inside args:
- `{ "i": 1 }`, `{ "f": 1.2 }`, `{ "s": "x" }`, `{ "b": true }`, `null`
- Arrays/Maps as value: `{ "array": [1, 2, 3] }`, `{ "map": { "k": 1, "k2": {"s":"v"} } }`
- Call/method/var literals: `{ "call": "fn", "args": [...] }`, `{ "method": "m", "object": {"var":"obj"}, "args": [...] }`, `{ "var": "name" }`
Diagnostics:
- When JSON cannot be mapped or arity mismatches, warnings are printed with `[macro][test][args]` and the specific test may be skipped unless `NYASH_TEST_ARGS_DEFAULTS=1` is set.
Notes
- Textual `quote(code)` is supported as a convenience via the existing parser, but `$name` placeholders should be built in AST templates directly.
- This API will evolve; treat as experimental until the stable `macro_box` interface is finalized.
## Expansion Safety (Depth/Cycle)
- The macro engine applies a bounded number of expansion passes and stops on fixpoint.
- Environment tuning:
- `NYASH_MACRO_MAX_PASSES` (default 32)
- `NYASH_MACRO_CYCLE_WINDOW` (default 8) — detect cycles across recent states
- `NYASH_MACRO_TRACE=1` — pass-by-pass logging

View File

@ -0,0 +1,26 @@
# Template → Macro Unification (Breaking Change)
Status: Active. Templates are now specified and executed via the Macro Engine. There is no separate “template pipeline”. This simplifies expansion order, diagnostics and tooling. The `--expand` flow shows both macro and template (pattern/unquote) transformations.
What changed
- Template constructs are expressed using the macro Pattern/Quote API.
- `$name` and `$...name` (variadic) placeholders are part of macro templates.
- Arrays/Maps are supported in both matching and unquote.
- There is no legacy, template-only expansion pass.
Authoring
- Programmatic API (Rust): `crate::macro::pattern::{TemplatePattern, OrPattern, AstBuilder}`.
- Textual quote is a convenience helper: `AstBuilder::quote("code here")` and unquote with variable bindings.
- Variadics:
- `$...tail` can appear at any position in call/array arguments.
- During match: the variable-length segment is bound to an `ASTNode::Program`.
- During unquote: the `Program`s statements are spliced into the argument list.
Tooling
- `nyash --expand --dump-ast file.nyash` shows pre/post expansion.
- Macro gate: `NYASH_MACRO_ENABLE=1`.
Compatibility
- This is a breaking change. Existing “template-only” extension points must adopt the macro engine.
- Guidance: treat templates as macro sugar; move any custom template processors to macro patterns.

View File

@ -155,3 +155,21 @@ DEBUG.startTracking()
DEBUG.trackBox(myObject, "説明")
print(DEBUG.memoryReport())
```
## Macro-based Test Runner (MVP)
Nyash provides a macro-powered lightweight test runner in Phase 16 (MVP).
- Enable and run tests in a script file:
- `nyash --run-tests apps/tests/my_tests.nyash`
- Discovers top-level `test_*` functions and Box `test_*` methods (static/instance).
- Filtering: `--test-filter NAME` (substring match) or env `NYASH_TEST_FILTER`.
- Entry policy when a main exists:
- `--test-entry wrap` → run tests then call original main
- `--test-entry override` → replace entry with test harness only
- Force apply: `NYASH_TEST_FORCE=1`
- Parameterized tests (MVP): `NYASH_TEST_ARGS_DEFAULTS=1` injects integer `0` for each parameter (static/instance tests).
- Exit code = number of failed tests (0 on success).
Notes
- The feature is behind the macro gate; CLI `--run-tests` enables it automatically.
- Future versions will add JSON-based per-test arguments and richer reporting.

150
docs/guides/user-macros.md Normal file
View File

@ -0,0 +1,150 @@
# User Macros (MacroBoxSpec) — Phase 2
Status: PoC complete; PyVM sandbox route wired. This guide explains how to author and run user macros in Nyash.
## Quickstart
- Register user macros (recommended minimal env):
- `NYASH_MACRO_ENABLE=1`
- `NYASH_MACRO_PATHS=apps/macros/examples/echo_macro.nyash`
- Run your program as usual (macro expansion happens once before MIR):
- `./target/release/nyash --backend vm apps/tests/ternary_basic.nyash`
Environment overview (recommended minimal set)
- `NYASH_MACRO_ENABLE=1`既定ON
- `NYASH_MACRO_PATHS=...`カンマ区切りのNyashマクロファイル
- `NYASH_MACRO_STRICT=1`(既定: 厳格)
- `NYASH_MACRO_TRACE=0|1`(開発用トレース)
Backward compat (deprecated)
- `NYASH_MACRO_BOX_NY=1` + `NYASH_MACRO_BOX_NY_PATHS=...` → 今後は `NYASH_MACRO_PATHS` を使ってね
## Philosophy
- Hybrid approach: built-in (Rust) for minimal/core derives; user macros in Nyash (PyVM) for flexibility.
- Deterministic, sandboxed execution: default denies IO/NET/ENV.
- Unified interface: AST JSON v0 + MacroCtx. Expansion occurs pre-MIR.
MacroCtx (MVP)
- Rust側に最小の `MacroCtx``MacroCaps` を用意将来のAPI統合のため
- フィールド/メソッドMVP:
- `MacroCtx::from_env()` → 環境からcapabilitiesを組み立て
- `ctx.gensym(prefix)` → 衛生識別子生成
- `ctx.report(level, message)` → 開発用レポート(標準エラー)
- `ctx.get_env(key)` → 環境取得(`NYASH_MACRO_CAP_ENV=1` のときのみ)
- 実行契約PoCランナーは `expand(json, ctx)` を優先し、失敗した場合は `expand(json)` にフォールバックします(後方互換)。
## Authoring a Macro
Create a Nyash file that defines `MacroBoxSpec` with a static `expand(json[, ctx])` returning a JSON string (AST JSON v0):
```nyash
static box MacroBoxSpec {
static function expand(json, ctx) {
// json: string (AST JSON v0)
// ctx: string (JSON; {caps:{io,net,env}} MVP). Optional for backward compatibility.
// return: string (AST JSON v0)
return json // identity for MVP
}
}
```
Example (repo): `apps/macros/examples/echo_macro.nyash`.
Editing template (string literal uppercasing)
- Example: `apps/macros/examples/upper_string_macro.nyash`
- Behavior: if a string literal value starts with `UPPER:`, the suffix is uppercased.
- Input: `print("UPPER:hello")` → Output: `print("HELLO")`
## Running your Macro
Register and run via env (simple):
```bash
export NYASH_MACRO_ENABLE=1
export NYASH_MACRO_PATHS=apps/macros/examples/echo_macro.nyash
# Run your program (macro expansion happens before MIR)
./target/release/nyash --backend vm apps/tests/ternary_basic.nyash
```
Selfhost pathNYASH_USE_NY_COMPILER=1での前展開開発用
```bash
NYASH_USE_NY_COMPILER=1 \
NYASH_MACRO_SELFHOST_PRE_EXPAND=1 \
NYASH_VM_USE_PY=1 \
./target/release/nyash --backend vm apps/tests/ternary_basic.nyash
```
Notes: 現状は PyVM ルートのみ対応。`NYASH_VM_USE_PY=1` が必須。
CLI プロファイル(推奨)
- `--profile dev`(既定相当: マクロON/厳格ON
- `--profile lite`マクロOFFの軽量モード
- `--profile ci|strict`マクロON/厳格ON
- 例: `./target/release/nyash --profile dev --backend vm apps/tests/ternary_basic.nyash`
Notes
- Built-in child route (stdin JSON -> stdout JSON) remains available when `NYASH_MACRO_BOX_CHILD_RUNNER=0`.
- Strict mode: `NYASH_MACRO_STRICT=1` (default) fails build on macro child error/timeout; set `0` to fallback to identity.
- Timeout: `NYASH_NY_COMPILER_TIMEOUT_MS` (default `2000`).
Testing
- Smoke: `tools/test/smoke/macro/macro_child_runner_identity_smoke.sh`
- Golden (identity): `tools/test/golden/macro/identity_user_macro_golden.sh`
- Golden (upper string): `tools/test/golden/macro/upper_string_user_macro_golden.sh`
- Golden (array prepend 0): `tools/test/golden/macro/array_prepend_zero_user_macro_golden.sh`
- Golden (map insert tag): `tools/test/golden/macro/map_insert_tag_user_macro_golden.sh`
- Negative (timeout strict fail): `tools/test/smoke/macro/macro_user_timeout_strict_fail.sh`
- Negative (invalid JSON strict fail): `tools/test/smoke/macro/macro_user_invalid_json_strict_fail.sh`
- Negative (invalid JSON nonstrict identity): `tools/test/smoke/macro/macro_user_invalid_json_nonstrict_identity.sh`
Array/Map editing examples
- Array prepend zero: `apps/macros/examples/array_prepend_zero_macro.nyash`
- Transforms every `{"kind":"Array","elements":[...]}` into one with a leading `0` literal element.
- Example input: `print([1, 2])` → Expanded: elements `[0, 1, 2]`.
- Map insert tag: `apps/macros/examples/map_insert_tag_macro.nyash`
- Transforms every `{"kind":"Map","entries":[...]}` by inserting the first entry `{k:"__macro", v: "on"}`.
- Example input: `print({"a": 1})` → Expanded entries: `[{"__macro":"on"}, {"a":1}]`.
## Inspect Expanded AST
```bash
./target/release/nyash --dump-expanded-ast-json apps/tests/ternary_basic.nyash
```
Outputs AST JSON v0 after expansion; use this for golden comparison.
## AST JSON v0 Schema
See `docs/reference/ir/ast-json-v0.md` for the minimal schema used in Phase 2.
## Troubleshooting
- Child timeout: increase `NYASH_NY_COMPILER_TIMEOUT_MS` or simplify macro code; strict mode fails fast.
- Invalid JSON from child: ensure `expand(json)` returns a valid AST JSON v0 string.
- No changes observed: confirm your macro is registered and the runner route is enabled.
- Capability denied: set caps explicitlyデフォルトは全OFF
- `NYASH_MACRO_CAP_IO=1` → IO系BoxFile/Path/Dir許可
- `NYASH_MACRO_CAP_NET=1` → NET系BoxHTTP/Socket許可
- `NYASH_MACRO_CAP_ENV=1``MacroCtx.get_env` 許可(将来拡張)
## Roadmap
- MacroCtx capabilities (io/net/env) expressed via nyash.toml per-macro.
- Diagnostics: JSONL tracing (`NYASH_MACRO_TRACE_JSONL`) and span/source maps.
- Golden tests for expanded JSON; strict mode as default.
## Capabilities (io/net/env)
Purpose: restrict sideeffects and ensure deterministic macro expansion.
- Default: all OFF (io=false, net=false, env=false)
- Behavior:
- io=false → no FileBox/FS access inside macro; AST JSON only
- net=false → no Http/Socket inside macro
- env=false → MacroCtx.getEnv disabled; child inherits scrubbed env
- Planned configuration (nyash.toml): see `docs/reference/macro/capabilities.md`
- PoC mapping: child is always `NYASH_VM_USE_PY=1`, `NYASH_DISABLE_PLUGINS=1`, timeout via `NYASH_NY_COMPILER_TIMEOUT_MS`
## Top-level static MacroBoxSpec (safety)
- 既定では無効(`NYASH_MACRO_TOPLEVEL_ALLOW=0`。Box宣言なしで `static function MacroBoxSpec.expand` を受理したい場合は `--macro-top-level-allow` を指定してください。