Stage A Software Benchmark
192 episodes · 2,140 turns · 36.7M tokens |
Four 10-module Python codebases (fin_ledger, clinical_ops, customs_tariff, grid_dispatch). Two parallel sprints (Draft + Repair) in isolated sandboxes with bash/file/web tools disabled, 82-line cap per module, AST-verified upstream imports, separate Sprint-1 smoke checks, and withheld production + compliance test suites. Matched m=10 core splits (Bank / Clin): θ1 bare 15%/8% → frozen 75%/100%; θ2 bare 63%/61% (29% at m=6) → frozen 100%/94%; θ3 4-domain mean 59% → 72% → 86% → 82% (coupled grid_dispatch 28% under frozen contracts); θ4 bare 5%/5% → frozen 74%/100%. |
Built-in integration scale weights: Downstream modules unlock higher workflow weights, so perfect output/worker rises across m = 1, 2, 4, 6, 10 (2.00, 2.40, 2.95, 3.40, 4.20). Fitted γ̂tail summarizes local scaling on m ∈ [4, 10], not unbounded m → ∞ asymptotics. Each cell runs 1 pass per domain (n = 2 matched core or n = 4 all domains). |
Controlled Specialist Proxies
H_code, H_spec, H_coord |
Holds the coding base model fixed at standard mid-tier (θ3) while varying endowments: H_code receives private algorithmic boundary tests; H_spec receives authoritative statutory disambiguation memos + policy smoke checks; H_coord runs frontier planner with Lsolo = 0 (specs & tickets only). |
Information & verification identification: Directly measures the value of private statutory clarity vs. algorithmic test suites vs. non-coding coordination—not biological human ergonomics or multi-year career adaptation. |
Upfront Contracts (τ3)
contract_ledger |
Supplies frozen Python function signatures and return-dict schemas before Sprint 1 plus automated smoke-test revert gates in Sprint 2. |
Exogenous contract creation: Measures execution once schemas are specified; does not price the human or compute labor required to author or maintain contracts as requirements change. |
Stage B1 Market Solver Normalized equilibrium |
Maximizes Marshallian surplus across Deep (Qsat=700) & Niche (Qsat=55) linear demands (P0=2.0) with scarce pool 6 H_code + 6 H_spec + 2 H_coord (wmin=2.15) and convex compute cost r(M) = r0[1 + α max(0, M/Mcap − μ)2] (r0=2.45, Mcap=95, μ=0.55, α=8). |
Headcount compute & homogeneous demand: Compute rental charges active clone seats M rather than raw token volume. Both demand pools share identical production functions, so PD* = PN* in B1; specialist retention under α > 0 comes from the marginal-compute price umbrella (P* = 1.425 > 0.632 break-even), not niche-specific differentiation. Shadow wages are central finite differences ∂W*/∂Hk. |
Stage B2 Live LLM CEOs
128 CEO turns (exploratory) |
Eight autonomous LLM CEOs across 4 model families simultaneously pick harnesses, market targets, and wage bids over 4 rounds (2 seeds × 4 rounds per compute regime). |
Separate calibration & cobweb swings: Saved B2 runs used an earlier parameterization (wmin=1.75, Mcap=150, α=5), included 15/115 parser-fallback turns (6/62 flat, 9/53 surge), and oscillate across the 4 simultaneous-move rounds rather than converging to a static rest point. |