The Science
Why AI changes who is deciding
Operion applies published research on how people use AI to the work engineering teams already do. This page shows what we take from the research, how it becomes a measurement, and which parts are still hypotheses.
- What the research shows
- How we measure it
- What is still unproven
A third system in the loop
Classic decision research describes fast intuition and slow deliberation. AI adds a third source of answers
01
System 1 and System 2
Fast, intuitive judgment and slow, deliberate reasoning. Experienced engineers build a quick sense that something is off in a change before they can say why.
02
System 3: artificial cognition
In tri-system theory (Shaw and Nave, 2025), AI is a third system outside the person. Its answer arrives already formed, before intuition or deliberation has done its work.
03
Offloading versus surrender
Offloading is using AI as a tool while keeping the judgment. Surrender is adopting its output as your own, unverified. The first is healthy. The second is the risk.
What the research found
In three preregistered experiments with 1,372 participants, people consulted the AI on more than half of trials even when they did not have to. When the AI was wrong they followed it most of the time, and having access to it raised their confidence regardless of whether it was right. Time pressure increased the effect. Incentives and feedback reduced it without removing it.
These are laboratory reasoning tasks. They show the mechanism. They do not show that the same rates hold in a code review.
Why it matters for engineering
A code reviewer, an incident commander, and a product lead each face the same conditions: fluent AI output, deadline pressure, and little time to form an independent view. Output metrics will not show the difference between a team that checked and a team that accepted.
Measuring it with signal detection
Signal detection theory separates what a person can detect from how willing they are to act on it
Sensitivity tells you whether a signal can be told apart from noise. The response criterion tells you how much evidence someone needs before they act. Two reviewers can approve the same share of changes and differ completely in sensitivity. We use this framing to look for shifts in how a team responds, not only how much it ships.
Dr. Anderson's own research in visual psychophysics, including work on how experts attenuate noise and how sensitivity varies across the visual field, is the reason we treat measurement design as the hard part, not the dashboard.
What we look for in the work
Each indicator below is a hypothesis drawn from the research. We say where each one stands
Review depth against change size
Whether review gets thinner as changes get larger, and how often changes merge without review.
Partial
Decision time near deadlines
Whether the time taken to approve or decide collapses as a deadline nears.
In development
Rework on fast changes
Reverts and follow-up fixes read against how quickly the original change moved.
In development
Separating AI-assisted work
Telling AI-assisted changes from the rest, so the comparison above is about AI and not just about speed.
In development
Lead time before outcomes
Whether these signals move before slipped projects or attrition. We have not backtested this yet and make no timing claim.
Validating
How we keep the measurement honest
A metric that becomes a target stops measuring anything, so the design works against that
Teams, never individuals
Findings are about team systems and flow. There are no individual scorecards to optimize against.
Evidence with every finding
Each finding shows the data behind it, how confident we are, and the research it draws on, so you can disagree with it.
Strength of evidence, stated
Frameworks differ in how well they have held up. We say when a source is contested and when a signal is an approximation.
Talk through what this looks like in your own work
Start with a conversation. We will say what we can measure on your data today and where we are still learning.
Led by Dr. Jen Anderson.