System 1 Models in Safety Research

What Jev and Laya Can and Cannot Do

Presenter

Amir Rafe, Ph.D.

Unit

AIT Lab

Date

2026-10-14

Hosted by

CE 7393

Tonight's question

Can a System 1 model carry a safety decision?

Can

Cannot

Verdict

To be completed by 7:00 PM.

Same report. Two kinds of answer.

Chat AI · writes

0 wordsyou still have to read it and judge it

Decision model · answers

Fall hazard?

0.99

Anyone injured?

0.04

0.2 snumbers your code can act on

Right: real Jev 1.13 answers on the synthetic scaffold report from Lab 1, recorded 10 Oct 2026. Left: a typical chat answer, written for this slide.

Attachment · Motion

Watch for

Both bars fill in one pass, while the generative model is still writing.

Note

Synthetic report and illustrative values.

The model scores. Your rule decides.

Five lineages traced in our survey of System One models (in preparation). Nothing here is new on its own; Jev put them behind one simple contract.

360 open models in nine days

Open System One checkpoints on Hugging Face by release date, from the census in our survey (in preparation). Originals are new models; derivatives are copies, conversions and fine-tunes.

Then the big companies arrived

Fastino blog post introducing GLiDE — **30 Sep** Fastino GLiDE, a model that thinks longer when unsure

Cloudflare blog post introducing Clef — **1 Oct** Cloudflare Clef, open weights

Microsoft announcement of Microsoft-Decision-1 — **9 Oct** Microsoft-Decision-1

Decision Index 0.3.1 leaderboard page — **Decision Index 0.3.1** · 10 Oct 2026

open weights weights not public Jev, size not disclosed

Big open models now reach Jev. Small ones, like Laya, are far behind.

EdgeEvals System One clinical leaderboard — **EdgeEvals System One** · 669 clinical decisions

Both boards put the same models near the top. Neither tests safety decisions.

So tonight we test them on safety data ourselves.

Routesend a defect note to the right crew

Guardstop an AI agent's unsafe action

Verifycheck another model's answer

Triagerank bridge alarms by urgency

Codeturn crash reports into variables

Tonight's question

Can a System 1 model carry a safety decision?

Seven short labs. You predict first, every time.

A workshop, not a lecture

Seven short labs. In each one you commit to a guess before you see any model output.

Every lab follows this procedure

PredictYou answer first, in Teams chat or a Teams poll.
RunWe run real code on screen and read the raw response.
RevealThe finding is stamped CAN or CANNOT and added to the tally.

Answer in

Teams chat

Votes

Teams polls

Code along

Colab, optional

Narrative Synthetic

Unit 1 northbound in light rain, left lane. Driver states she looked down at her phone for directions, looked up, and traffic had stopped. Unit 1 struck the rear of Unit 2. No injuries reported.

Was the driver distracted? Did the weather contribute?

Answer in Teams chat

distracted 90, weather 20

For each question, how likely the answer is yes, in percent.

Time

0:20

You read a report and returned two probabilities.

That is exactly the job of the decision models you just met. A typed answer and how sure, nothing else.

Recorded run · jev-1.13.0 · 10 Oct 2026

Model's answer

Was the driver distracted?P(yes)

0.97

Did the weather contribute?P(yes)

0.32

Compare with the chat. Where did the class disagree most?

Lab 1

Anatomy of one call

One construction report, three typed questions, two models. Same request, very different answers.

Procedure

PredictWill an open model on a regular computer agree with the hosted one?
RunThe same three questions go to Laya and to Jev.
RevealWe compare the answers with what the report says.

Time

8 min

Colab

Cells 1 to 3

Evidence

arXiv 2610.00346

Attachment A · laya 0.4.2


from laya import Router

router = Router()   # downloads the open checkpoint once

state = ("Worker on level 3 scaffold. East guardrail missing. "
         "Worker slipped, harness held, no injury.")
questions = {
  "fall_hazard":  {"type": "noul",
     "instructions": "Does the report describe a fall hazard?"},
  "harness_held": {"type": "noul",
     "instructions": "Did fall protection stop the fall?"},
  "injury":       {"type": "noul",
     "instructions": "Was anyone injured?"},
}
result = router.predict(state, questions)

Recorded run · 0.17 s

Response · Laya default checkpoint

fall_hazardP(yes)

0.22

harness_heldP(yes)

0.61

injuryP(yes)

0.55

Read the report again. Which of these three is wrong?

Accuracy on typed decisions and intents (arXiv 2610.00346)

Model	Typed decisions	CLINC-150
DeBERTa-v3-large, fine-tuned on labels	0.793	0.953
Gemma-4-31B, generative, no labels	0.741	0.942
Jev 1.13, no labels	0.732	0.913

Swap only the names "yes" and "no", same descriptions

Model	Flips per 100 · fewer is better
Kev-9B	6.5
Jev 1.13	50.5
Laya English	91.3

Where labels exist, a fine-tuned classifier still wins. Without labels, a System 1 model is a strong, fast start.

Our benchmark, arXiv 2610.00346. Eight decision checkpoints in six families plus four open generative models and four classifier baselines, eight datasets.

These models can read a report and answer your yes or no questions in under a second.

Reviewed by

Can

These models cannot be trusted until you test them on your own reports.

Reviewed by

Cannot

Lab 2

Design the schema

You write the questions. We add gates, run them on one report, and read what comes back.

Procedure

PredictPost one typed question in Teams chat.
RunYour questions go into the schema and run live.
RevealWe read the probabilities and stamp the verdict.

Time

10 min

Colab

Cell 4

Evidence

arXiv 2609.24052

Narrative Synthetic

Unit 2 was stopped at the light on FM 1626. Driver of Unit 1 stated he felt dizzy and blacked out before the crash. Witness stated Unit 1 did not brake. Driver of Unit 1 transported to hospital by EMS.

Type · noul

Yes or no

Did the driver have a medical episode?

Type · choice

Pick one option

What kind of episode? Seizure, blackout, diabetic, cardiac, other.

Type · score

An ordinal level

How detailed is the narrative, from 0 to 3?

Your turn · Teams chat

Post one yes or no question you would ask of this report.

For example, Was the driver of Unit 1 taken to the hospital?

1:00

Attachment · Motion

Watch for

A screen question opens or shuts a gate. Follow-up questions run only behind an open gate.

Why gate

Fewer questions per report, and no detail asked of reports where it cannot apply.

Attachment B · crash_factors_v1.json


"medical_episode": {
  "type": "noul",
  "instructions": "The narrative states that a driver had a
    medical episode, such as fainting or a seizure."},
"medical_type": {
  "type": "choice",
  "instructions": "What kind of medical episode does the
    narrative describe?",
  "criteria": {
    "seizure": "...", "loss_of_consciousness": "...",
    "diabetic": "...", "cardiac": "...", "other": "..."}},
"gates": {
  "medical_type": "medical_episode",
  "drug_class":   "drug_involved", ...}

The full schema

Questions	27
Yes or no (presence)	16
Choice (categorical)	8
Score (ordinal)	3
Behind a gate	7

A gate opens when the screen question returns a probability above 0.5.

Instruction text shortened for the slide. Variables include medical episode, drugs, alcohol, hydroplaning, vehicle defect, fatigue, aggression, phone use, wrong way, animal, pregnancy.

Texas crash narratives, 2017 to 2025 (arXiv 2609.24052)

Narratives in scope	5,018,080
Screened with 10 questions	499,500
Coded with all 27 questions	195,857
Blinded human judgments	2,416

Agreement with human labels (F1)

Jev 1.13	0.908
Frontier generative model A	0.967
Frontier generative model B	0.885

Injury and fatal crashes per year attributed to nine factors

Coded fields

28,057

Plus narratives

+10,747

codedadded from narratives

Phone use rises by 182 percent. Four of nine factors change rank.

Code police crash narratives at state scale with a gated schema.

Evidence

arXiv 2609.24052. Jev 1.13 on 195,857 Texas narratives. F1 of 0.908 against 2,416 blinded human judgments.

Reviewed by

Can

Lab 3

The trust meter

A probability is only useful if 0.9 means right nine times in ten. We check that, and fix it when it fails.

Procedure

PredictTeams poll on what 0.90 really means.
RunReliability curves from three of our studies.
RevealHow many reports a person still has to read.

Time

10 min

Colab

Cell 5

Evidence

arXiv 2609.24052
arXiv 2610.00346
Pedestrian study

The model says 0.90 on a hundred reports.

How many does it get right?

Teams poll 2 · single choice

About 90
About 75
About 50
Cannot know until we measure

Launch poll 2 from the Polls tab now.

Stated confidence against observed accuracy

Source

Top bin, stated

0.00

mean confidence

Top bin, observed

0.00

share correct

Attachment C · Platt scaling


import numpy as np
from sklearn.linear_model import LogisticRegression

# p: model probabilities, y: human labels (0 or 1)
p = np.clip(p, 1e-4, 1 - 1e-4)
z = np.log(p / (1 - p)).reshape(-1, 1)    # model log-odds
platt = LogisticRegression().fit(z, y)     # learn slope and shift
p_new = platt.predict_proba(z)[:, 1]        # recalibrated

arXiv 2609.24052 · calibration error

As returned by Jev	0.0231
After Platt scaling, held out	0.0069

3.3 times smaller error, from labels you already need for the audit.

Recalibration fixes miscalibrated numbers. It cannot add information the model never had, which is why the pedestrian curve stays poor.

To reach 90 percent precision, what share of flagged reports must a person still read?

Variable	Flags per year	Judged against coded fields	Judged against human labels
Alcohol	20,262	100%	5.1%
Hydroplaning	13,114	100%	0.0%
Wrong way	7,958	100%	72.2%
Fatigue	7,798	100%	0.1%
Medical episode	5,880	100%	14.4%

The coded field is a poor yardstick. Measured against it, agreement looks worse by a median kappa of 0.26.

Be trusted on the model card's word. Calibration changes with the model and the task.

Evidence

Crash narratives, well calibrated tails (arXiv 2609.24052). Pedestrian choices, stated 0.98 and right less than half the time. Typed decisions, Laya confidence carries little signal (arXiv 2610.00346).

Reviewed by

Cannot

Lab 4

Two readers, one crash

The coded record and the officer's narrative describe the same crash. They do not always agree.

Procedure

PredictWhat did the coded record miss?
RunThe model reads the narrative.
RevealHow often the form misses it across Texas.

Time

7 min

Colab

Not needed

Evidence

arXiv 2610.00213
arXiv 2610.10321

Coded record Synthetic

Crash type	Rear end
Severity	Possible injury
Contributing factor 1	Followed too closely
Contributing factor 2	None
Distraction coded	No

Narrative Synthetic

Unit 1 was traveling west on Loop 410 in moderate traffic. Driver of Unit 1 stated he was reading a text message when traffic slowed ahead. Unit 1 struck the rear of Unit 2. Driver of Unit 2 complained of neck pain.

Model reads the narrative · recorded run · jev-1.13.0

Phone use stated in the narrativeP(yes)

0.98

Injury or fatal crashes, 2017 to 2025 · coded count and narrative estimate (arXiv 2610.10321)

Phone use

15,074

Medical episode

28,345

Hydroplaning

32,996

Fatigue

24,880

Animal

20,195

coded in the crash recordnarrative estimate above the coded count

The narrative shows about twice the phone use the form records. A second human check agreed on all fifteen factors.

The crash form has no field for pregnancy. The narrative sometimes says it.

Our pregnancy study (arXiv 2610.00213) read every narrative flagged by a keyword screen and estimated what the screen missed.

Texas, 2017 to 2025 (arXiv 2610.00213)

Narratives in scope	5,018,079
Crashes documenting a pregnancy	5,467
95 percent interval	5,306 to 5,577
Core keyword hits the model confirmed	97.9%

Yet that is only about 3 percent of the pregnant drivers vital statistics predict, and mostly the injured ones.

Our estimates have a 22 percent margin. Which spends effort better?

Root mean square relative margin across factors (arXiv 2610.10321)

Now

22.0%

Model reads all

21.2%

400 more labels

16.2%

Reading all five million narratives barely helps. Four hundred more human-coded narratives do.

Recover factors the coded form misses, then size them with human labels.

Evidence

About twice the coded phone use, confirmed by a second human check (arXiv 2610.10321). Pregnancy found in 5,467 crashes on a form that never asks (arXiv 2610.00213).

Reviewed by

Can

Lab 5

Be the pedestrian

Can a model that reads text stand in for a person walking in a crowd?

Procedure

PredictTeams poll on which way this pedestrian goes.
RunThe model answers the same described scene.
RevealThree thousand real decisions decide the verdict.

Time

8 min

Data

Jülich corridors

Evidence

Pedestrian study
in preparation

Attachment D · state sent to the model


{"your_pace": "walking briskly",
 "your_destination": "straight ahead",
 "crowding": "open space around you",
 "wall_on_your_left": "far away",
 "wall_on_your_right": "one step away",
 "space_directly_ahead": "clear for several steps",
 "people_nearby": [
   "ahead to your left, a few steps away:
    pulling away from you, walking the same
    way at your pace", "..."]}

Teams poll 3 · which way next?

Hard left
Left
Straight
Right
Hard right

Model · jev-1.13.0 · and the measured path

Straight

0.85

The person did go straight.

Attachment · Motion

Watch for

Spacing, speed and body contact do not survive the translation into a sentence.

Note

Illustrative values, not a model run.

Direction choice, one-way corridor, 1 s ahead · log loss, lower is better (pedestrian study, in preparation)

Logistic model on the same words	0.966
Always guess the base rates	1.410
Jev 1.13, zero shot	1.923

Pushing at a bottleneck, four categories · accuracy

Repeat last second's rating	0.956
Logistic model on the same words	0.70
Jev 1.13, zero shot	0.53

It says 0.98 and is right less than half the time. It never gets a hard turn right.

Adding its answers to the simple model improves it by at most 3.12 percent, and makes every pushing task worse.

Jülich 2013 corridors and the 2018 Wuppertal bottleneck experiment. About 3,000 decisions per corridor setting, 3,640 rated person-seconds for pushing.

Stand in for a pedestrian. When physics is turned into words, a simple model on the same words wins.

Evidence

Pedestrian study, in preparation. Zero-shot Jev 1.13 on Jülich corridors and a bottleneck experiment. Laya was not tested here.

Reviewed by

Cannot

Construction Synthetic

Trench report

8 ft excavation at Pier 3, no shoring or trench box. Laborer entered to clear a pump hose.

Cave-in hazard described?

Bridges Synthetic

Inspection note

Girder G3, 0.5 mm diagonal crack near the east bearing, rust staining, bearing appears frozen.

Needs a site visit before the next cycle?

Water Synthetic

Main break

12-inch cast iron main break near hydrant 44. Roadway undermined and sagging.

Hazard to traffic described?

Buildings Synthetic

Fire report

Kitchen fire, four-story apartment. No sprinkler system. One resident treated for smoke.

Suppression missing or failed?

Your turn · Teams chat

In the chat, name a card and ask one yes or no question about it.

For example, Fire report. Did the smoke detectors activate?

0:45

Check every box

The state is written by peopleReports, notes, logs. Not physics turned into words.
Questions are typed and gatedFixed options, asked only when they apply.
This model was tested on your dataCheckpoint and wording both change answers.
Calibration is measuredOn held-out human labels, then recalibrated.
A human review budget is setAgainst human labels, not the coded field.

Tonight's question · share of verdicts

Can a System 1 model carry a safety decision?

Can

0.00

Cannot

0.00

Verdict

Strong on what people wrote down. Weak where physics is squeezed into words. Never trusted before it is measured.

The six studies behind tonight · Rafe and Das, 2026 · arXiv numbers

2609.24052	Calibrated decisions at scale. Crash narratives into calibrated variables.
2610.00213	Pregnancy in crash narratives. A variable no form records.
in prep.	Calibrated decisions without crowd sense. A boundary study.
2610.00346	Benchmarking System One models for decision gates.
in prep.	A survey and census of System One models.
2610.10321	Estimating uncoded crash factors with Kumo Tabular and Jev.

Notebook

The Colab notebook from the chat runs the labs with Laya on a free CPU. No key needed.

Follow the field

Decision Index on Hugging Face (multimodalart). EdgeEvals System One. Our benchmark, arXiv 2610.00346.

Contact

Amir Rafe, Ph.D.
AIT Lab, Ingram School of Engineering
Texas State University

Questions

What would you ask a System 1 model in your own work?

Reviewed by

Measured