laya-as-judge 0.4 (Laya-Omega): beating Gemini Decision 1 across all 9 benchmarks, Top-1 ECE (0.0234), and total cost ($0.77)

radeksixi · 13 November 2026 · Used in production with sixi.ai · source, calibration math & benchmark suite

Autoregressive "LLM-as-a-Judge" pipelines burn dollars generating paragraphs of JSON when software actually needs a calibrated distribution over typed outcomes (noul boolean, choice, or ordinal score). Standalone local encoders (Laya 421M/322M) fix latency and output tokens, but hit an obvious ceiling on raw screenshots, audio clips, video streams, and multi-hop domain reasoning. laya-as-judge 0.4 (Laya-Omega)—the open-source evaluation engine built and used alongside sixi.ai—combines local CUDA encoders and disjoint Test-Time Training with a selective two-stage Gemini Interactions API cascade and out-of-fold simplex meta-calibration. Across 2,250 real Hugging Face questions (N = 250 per benchmark, seed = 42, zero test leakage), it beats the published Gemini Decision 1 scoreboard on all 11 rows: every single quality benchmark, pooled Top-1 ECE, and total dollar cost.

Architecture: three tiers instead of one hammer

Calling a frontier reasoning model on every row blows the $0.88 budget; keeping everything on a 421M local encoder fails multimodal tasks and nuanced multilingual STEM. OmegaDecisionEngine treats evaluation as a risk-controlled routing problem:

• Tier 1 — Local CUDA encoders at $0.00 (TorchBackend + VegaBackend): Every text item first passes through local convaiinnovations/laya and vega-U (0.8B hybrid linear-attention backbone) on GPU. For benchmarks with public training splits (Amazon ESCI, When2Call, PubMedQA), lightweight Ridge / Logistic Test-Time Training (TTTHead) heads are fitted strictly on disjoint training data (train ∩ test = ∅).
• Tier 2a — Non-thinking fast cloud verifier (gemini-3.5-flash-lite): Items whose local confidence or margin falls below threshold—plus high-resolution UI grounding (Mind2Web Multimodal)—escalate via the google-genai Interactions API (store=False) with strict JSON Schema output and zero thinking-token overhead.
• Tier 2b — Selective reasoning escalation (gemini-3.8-flash): Only the hardest tail of Stage-2a items (ranked by top-2 probability margin p_(1) - p_(2)), plus dense structured/multimodal packs (JevBench at 92.80%, MMAU, and Video-MME at 78.40%), escalate to gemini-3.8-flash using multi-item packed prompts (ITEM_i0..ik) that amortize shared rubrics and thinking tokens by 3×–12×.

Multimodal grounding without subtitle shortcuts

Three of the nine benchmarks require non-text perception, and each had a specific token-to-signal bottleneck:

• Video-MME without subtitles (78.40% vs 69.1%): No YouTube subtitles are touched. Each MP4 is streamed via HTTP range requests, decoded into 18 chronological frames stamped with MM:SS timestamps on a single 6×3 storyboard image (halving vision input tokens compared to multi-sheet layouts), and paired with local GPU faster-whisper speech transcription.
• Mind2Web Multimodal: Full-page web screenshots are cropped around candidate DOM bounding boxes and rendered with color-coded Set-of-Mark overlays [A]..[E] alongside compact interactive attributes.
• When2Call (80.80% vs 79.9%): Deterministic schema pre-auditing checks whether candidate tool calls exist in available_tools and whether required JSON Schema parameters are present in the user prompt before the model scores the four tool-routing actions.

Solving Amazon ESCI's rare-class trap (59.70% Macro-F1 vs 58.7%)

Amazon ESCI is scored by unweighted 4-class Macro-F1 across Exact (~65%), Substitute (~22%), Irrelevant (~10%), and Complement (~3.2%). A model can hit 71% raw accuracy while scoring 52% Macro-F1 simply by under-predicting Complement. Worse, cloud models emit near-one-hot posteriors ([0.01, 0.01, 0.02, 0.96]), which drown out small linear blends with local models. Combining log-probabilities from local Vega-U TTT (trained on disjoint US ESCI training queries) with Tier-2 posteriors and tuning class multipliers on held-out train predictions lifts rare-class recall cleanly without distorting Exact precision.

Simplex meta-calibration: getting Top-1 ECE from 0.0298 to 0.0234

Post-hoc scalar temperature scaling cannot simultaneously calibrate $0.00 local encoders and structured JSON probabilities from two different cloud models across binary, 3-way, 4-way, and 5-way questions. MetaCalibrator fits a 10-fold strictly out-of-fold logistic calibrator over logit(p_max), top-2 margin, normalized Shannon entropy, log(K), and a per-tier interaction slope, then reconstructs the full probability vector on the simplex while preserving 100% of Top-1 choices.

What did not work

• Over-packing gemini-3.5-flash-lite: Packing 8–10 items per call cuts cost by 4× on thinking models (gemini-3.8-flash) because one thinking trace is amortized across the batch. On gemini-3.5-flash-lite (which emits 0 thought tokens), pack_size=10 saved almost nothing while diluting cross-item attention on When2Call from 90.0% down to 62.5%. Keeping pack_size=1..3 on non-thinking calls and pack_size=4..12 on thinking calls fixed both accuracy and cost.
• Naive runner-up simplex clamping: Our first argmax-preserving probability reconstructor enforced q ≥ r_max / (1 + r_max) + ε keeping fixed runner-up ratios. For uncertain 4-way predictions where one runner-up held most of the residual mass, that floor artificially clamped calibrated confidence upward toward 0.48, stalling pooled ECE at 0.0251 (missing < 0.0250 by 0.0001). Blending runner-up shares smoothly toward uniform when q approaches the true simplex floor (1 + (K-1)ε)/K eliminated the upward bias while keeping every argmax invariant.
• Sending cartoon images to New Yorker Caption: The benchmark already supplies official human-written location, scene_description, and comic_element fields. Passing inline JPEGs burned 285k extra tokens (+$0.21) for zero accuracy gain (93.6% with image vs 93.3% text-only).
• Pretty-printed float JSON output bloat: Output tokens cost 5×–8× more than input tokens. Emitting multi-line 16-digit float JSON (0.010000000000000002) with verbose question keys burned 55 output tokens per question ($0.38 across 2,250 items). Switching the Interactions API schema to single-line integer-percentage arrays ("p": [5, 85, 10] summing to 100) under 1-token keys q0..qN plus resolution="medium" on UI/storyboard images cut output tokens 2.75× to 20.0 tokens/item with zero calibration loss.

All 2,250 questions were evaluated on real Hugging Face splits with seed=42, strict train/test disjointness (train ∩ test = ∅ across TTT heads, ESCI stacker/multipliers, JevBench precedent cases, and 10-fold out-of-fold ECE), and micro-dollar token accounting across every API call. laya-as-judge powers fast evaluation and deterministic-first guardrails alongside sixi.ai.

Source, Apache-2.0: github.com/radeksixi/laya-as-judge · Ecosystem: sixi.ai · More writing