62 lines
2.8 KiB
Markdown
62 lines
2.8 KiB
Markdown
|
# Strings Blueprint — UTF‑8 First, Bytes Separate
|
|||
|
|
|
|||
|
|
Status: active (Feature‑Pause compatible)
|
|||
|
|
Updated: 2025-09-21
|
|||
|
|
|
|||
|
|
Purpose
|
|||
|
|
- Unify string semantics by delegating StringBox public APIs to dedicated cursor boxes.
|
|||
|
|
- Keep behavior stable while making codepoint vs byte decisions explicit and testable.
|
|||
|
|
|
|||
|
|
Pillars
|
|||
|
|
- Utf8CursorBox (codepoint-oriented)
|
|||
|
|
- length/indexOf/substring operate on UTF‑8 codepoints.
|
|||
|
|
- Intended as the default delegate for StringBox public APIs.
|
|||
|
|
- ByteCursorBox (byte-oriented)
|
|||
|
|
- length/indexOf/substring operate on raw bytes.
|
|||
|
|
- Use explicitly for byte-level parsing or binary protocols.
|
|||
|
|
|
|||
|
|
Delegation Strategy
|
|||
|
|
- StringBox delegates to Utf8CursorBox for core methods: length/indexOf/substring.
|
|||
|
|
- Provide conversion helpers: toUtf8Cursor(), toByteCursor() (thin wrappers).
|
|||
|
|
- Prefer delegation over inheritance; keep “from” minimal to avoid API ambiguity.
|
|||
|
|
|
|||
|
|
API Semantics
|
|||
|
|
- indexOf: define two flavors via the box boundary.
|
|||
|
|
- StringBox.indexOf → Utf8CursorBox.indexOf (CP-based; canonical)
|
|||
|
|
- ByteCursorBox.indexOf → byte-based; opt‑in only
|
|||
|
|
- substring: follow the same split (CP vs Byte). Do not mix semantics.
|
|||
|
|
- Document preconditions for indices (out‑of‑range clamped/errored per guide).
|
|||
|
|
|
|||
|
|
Implementation Plan (staged, non‑breaking)
|
|||
|
|
1) Provide MVP cursor boxes (done)
|
|||
|
|
- apps/libs/utf8_cursor.nyash
|
|||
|
|
- apps/libs/byte_cursor.nyash
|
|||
|
|
2) Delegate StringBox public methods to Utf8CursorBox (internal only; behavior unchanged)
|
|||
|
|
- Start with length → indexOf → substring
|
|||
|
|
- Add targeted smokes for edge cases (multi‑byte CP, boundaries)
|
|||
|
|
3) Replace ad‑hoc scans in Nyash scripts with cursor usage (Mini‑VM/macros)
|
|||
|
|
- Migrate internal scanners (no external behavior change)
|
|||
|
|
4) Introduce ByteCursorBox only where byte‑level semantics are required
|
|||
|
|
- Keep call sites explicit to avoid ambiguity
|
|||
|
|
|
|||
|
|
Transition Gate (Rust dev only)
|
|||
|
|
- Env `NYASH_STR_CP=1` enables CP semantics for legacy byte-based paths in Rust runtime (e.g., StringBox.length/indexOf/lastIndexOf).
|
|||
|
|
- Default remains byte in Rust during the feature‑pause; PyVM follows CP semantics. CI smokes validate CP behavior via PyVM.
|
|||
|
|
|
|||
|
|
Related Docs
|
|||
|
|
- reference/language/strings.md — policy & scope
|
|||
|
|
- guides/language-core-and-sugar.md — core minimal + sugar
|
|||
|
|
- reference/language/EBNF.md — operators (! adopted; do‑while not adopted)
|
|||
|
|
- guides/loopform.md — loop normalization policy
|
|||
|
|
|
|||
|
|
Box Foundations (string-related)
|
|||
|
|
- Utf8CursorBox, ByteCursorBox
|
|||
|
|
- StringExtBox (trim/startsWith/endsWith/replace/split)
|
|||
|
|
- StringBuilderBox (append/toString)
|
|||
|
|
- JsonCursorBox (lightweight JSON scanning helpers)
|
|||
|
|
|
|||
|
|
Testing Notes
|
|||
|
|
- Keep PyVM as the reference execution path.
|
|||
|
|
- Add smokes: CP boundaries, mixed ASCII/non‑ASCII, indexOf not found, substring slices.
|
|||
|
|
- Avoid perf work; focus on semantics + observability.
|