🔍 Research: GPT-5-Codex capabilities and GitHub PR integration

## Summary
Investigated OpenAI's new GPT-5-Codex model and Codex GitHub PR review integration capabilities.

## GPT-5-Codex Analysis

### Benchmark Performance (Good)
- SWE-bench Verified: 74.5% (vs GPT-5's 72.8%)
- Refactoring tasks: 51.3% (vs GPT-5's 33.9%)
- Code review: Higher developer ratings

### Real-World Issues (Concerning)
- Users report degraded coding performance
- Scripts that previously worked now fail
- Less consistent than GPT-4.5
- Longer response times (minutes vs instant)
- "Creatively and emotionally flat"
- Basic errors (e.g., counting letters incorrectly)

### Key Finding
Classic case of "optimizing for benchmarks vs real usability" - scores well on tests but performs poorly in practice.

## Codex GitHub PR Integration

### Setup Process
1. Enable MFA and connect GitHub account
2. Authorize Codex GitHub app for repos
3. Enable "Code review" in repository settings

### Usage Methods
- **Manual**: Comment '@codex review' in PR
- **Automatic**: Triggers when PR moves from draft to ready

### Current Limitations
- One-way communication (doesn't respond to review comments)
- Prefers creating new PRs over updating existing ones
- Better for single-pass reviews than iterative feedback

## 'codex resume' Feature
New session management capability:
- Resume previous codex exec sessions
- Useful for continuing long tasks across days
- Maintains context from interrupted work

🐱 The investigation reveals that while GPT-5-Codex shows benchmark improvements, practical developer experience has declined - a reminder that metrics don't always reflect real-world utility\!
This commit is contained in:
Selfhosting Dev
2025-09-16 16:28:25 +09:00
parent 47f4ca0e44
commit 63c8fda808
41 changed files with 854 additions and 146 deletions

View File

@ -29,9 +29,22 @@ def lower_newbox(
vmap: Value map
resolver: Optional resolver for type handling
"""
# Use NyRT shim: nyash.env.box.new(type_name: i8*) -> i64
# Use NyRT shim: prefer birth_h for core boxes, otherwise env.box.new_i64x
i64 = ir.IntType(64)
i8p = ir.IntType(8).as_pointer()
# Core fast paths
if box_type in ("ArrayBox", "MapBox"):
birth_name = "nyash.array.birth_h" if box_type == "ArrayBox" else "nyash.map.birth_h"
birth = None
for f in module.functions:
if f.name == birth_name:
birth = f
break
if not birth:
birth = ir.Function(module, ir.FunctionType(i64, []), name=birth_name)
handle = builder.call(birth, [], name=f"birth_{box_type}")
vmap[dst_vid] = handle
return
# Prefer variadic shim: nyash.env.box.new_i64x(type_name, argc, a1, a2, a3, a4)
new_i64x = None
for f in module.functions: