Project chronology
The record distinguishes planning, implementation, evaluation, and public presentation.
The named STRATA program is documented by this date. Planning completed first; the CPU contract implementation then began under explicit evidence gates.
The first model-backed STRATA evaluation ran, followed by the controlled comparisons and characterization reported on this page.
September 2026
The case study entered the public portfolio during the September release cycle.
The real need
Make several reliable decisions from shared evidence without hiding the reasoning inside a paragraph.
Operational work often starts with the same body of evidence—a request, incident record, policy, test result, or maintenance history—and needs several bounded answers. Who owns it? What evidence is present or missing? Which condition applies? When should a person step in?
STRATA tests whether a compact local model can answer those questions as defined choices with visible score distributions. The goal is not to replace judgment. It is to produce results that can be compared, challenged, routed, and reviewed consistently.
The approach
Define the decision, compare honest baselines, challenge the evidence, and keep a person responsible.
Bound the question
Specify the shared evidence, the independent questions, allowed choices, missing-information behavior, and what failure must look like.
Score and control
Compare typed model decisions with matched generation and transparent executable controls before promoting the more complex path.
Challenge the evidence
Use altered, removed, contradictory, and injected evidence to expose shortcuts, instability, and unwarranted confidence.
Accept, revise, or stop
Combine quality, probability behavior, performance, software verification, and human review into an explicit next decision.
Latest evidence
The most useful result is knowing which ideas deserve to move forward.
1,000 questions
Development evaluation covered eight task families under the frozen comparison.
79.4% vs. 100%
Model scoring and compact generation tied at 79.4%; the transparent executable control solved the synthetic workload.
43.81% bounded reduction
Qualified serial state reuse reduced observed latency on one fixed short-prefix, 16-question fixture while preserving every top choice.
ECE 0.06645 → 0.02439
Temperature scaling improved aggregate probability metrics without changing decisions; no operating threshold was promoted.
Shortcut audit stopped the run
A canary benchmark was revised before model compute because simple controls recovered most or all provisional labels.
85 / 85 checks
The maintained CPU verification suite passes, along with formatting and lint checks.
What changed
Ordinary code won the templated task, keeping the solution simpler.
Direct candidate scoring and matched compact generation each reached 79.4% accuracy and made the same choices across all 1,000 development questions. A transparent rule and lexical control reached 100% on that synthetic workload. STRATA therefore preserved the simpler path for that workload and reserved model evaluation for harder problems where semantic reasoning can add value.
That is a successful engineering outcome: the program avoided unnecessary model cost and uncertainty while preserving the evaluation harness for more demanding decisions.
Bounded performance
Serial reuse qualified on one measured fixture.
After warmups, each mode completed 102 whole requests on a fixed 16-question, short-prefix fixture. Singleton direct scoring averaged 2.1217 seconds, right-padded batched scoring averaged 2.3088 seconds, and the conditionally qualified shared-serial path averaged 1.1922 seconds with full top-choice agreement.
The fixture recorded a 43.81% reduction versus direct scoring. The shared-parallel experiment also exposed BF16 batch-shape numerical instability, so the project retained the serial path and moved parallel reuse into a future numerical-stability investigation.
Probability characterization
Aggregate probability gains established the next confidence work.
On a one-time 1,000-question final test, global temperature scaling improved negative log loss from 0.55228 to 0.52923, Brier score from 0.31469 to 0.30520, and expected calibration error from 0.06645 to 0.02439 while preserving every decision.
Mixed subgroup effects identified the next validation need: subgroup-aware confidence testing before defining acceptance, escalation, or automation thresholds.
Benchmark correction
Catch authored shortcuts before spending compute on the wrong question.
A 32-record incident canary covered eight licensed public incident families with clean, evidence-removed, contradictory, and prompt-injected variants. A transparent rule recovered all 32 provisional joint decisions. Leave-one-source-group-out lexical classification recovered 30 of 32 variants and all 32 evidence-status labels; after marker ablation, it still recovered 24 of 32 variants and every evidence-status label.
The project preserved compute by identifying the shortcut before another model run. The first version remains useful for human-usability review, while the second version will use a stricter evidence-authoring standard and broader source diversity for model validity.
Human review
Separate technical validity from whether the result is understandable and useful.
Blinded review packets are prepared for technical reviewers, general users, general managers, and an industrial-operations subject-matter reviewer. The intake preserves packet identity and supports disagreement-focused adjudication without exposing one reviewer to another reviewer’s answers.
Independent human review is the next active stage and will establish usability, explanation quality, and domain interpretation across those audiences.
Industrial direction · design target
An advisory reliability layer that helps people narrow the next investigation.
A future lane would rank likely fault or component domains from PLC, SCADA, DCS, sensor, instrumentation, interlock, motor, bearing, conveyor, and condition-monitoring evidence. It would identify missing information, estimate coarse maintenance windows, abstain or escalate when evidence is insufficient, and later support deterministic notifications.
The designed role is advisory: organize evidence, rank investigation paths, identify missing information, and keep operating and safety decisions with responsible people. Hardware integration, notification delivery, and safety-system participation are later validation stages.
My contribution
Technical direction, promotion gates, and the judgment to reject a weak path.
I directed and scoped the program, framed the decisions it must support, set the architecture and tradeoff boundaries, designed the evaluation and promotion gates, challenged weak assumptions, prioritized safety and evidence tests, characterized bounded performance, rejected an unstable parallel path, and stopped a shortcut-prone benchmark before additional compute was committed.
AI agents assisted with implementation and analysis. My accountable contribution is the problem framing, requirements, technical direction, corrections, acceptance standard, and translation of results into the next decision.
Current state
A research-characterized prototype with a defined next evaluation stage.
STRATA’s current evidence includes typed scoring, matched controls, bounded state reuse, probability characterization, adversarial variants, maintained software checks, and prepared blinded-review workflows.
The next stage is second-version incident evidence with broader source diversity, followed by independent blinded review and a decision on the next model evaluation.
Skills demonstrated
What this project brings together.
- AI product direction
- Decision-workflow design
- Requirements engineering
- Controlled model evaluation
- Baseline and leakage testing
- Probability characterization
- Performance characterization
- Human-review design
- Industrial advisory scoping
- Python
- PyTorch
- Hugging Face Transformers
- NVIDIA GB10