TL;DR

Amazon security VP warns that human-in-the-loop AI oversight fails quickly because attention fades over time. Google, Microsoft, and IBM share similar concerns.

Amazon security leaders are questioning a core idea in AI governance. Eric Brandwine, VP and distinguished engineer at Amazon Security, told The Register that human oversight is not the reliable safeguard many assume.

"Humans are not terribly consistent," Brandwine said. "Human-in-the-loop is not necessarily the gold standard."

His view stems from the idea of normalization of deviance, a topic he addressed in a 2017 AWS re:Invent talk. The concept explains how repeated shortcuts become accepted when no major issues arise immediately.

Brandwine used emergency rooms as an example. A nurse responds to every alarm on day one.

After repeated false alarms without bad outcomes, focus slips. A genuine crisis can then be overlooked.

"Literally, someone's life is on the line, and people still struggle to maintain discipline," Brandwine said. "That is the human condition."

He extended this to AI agent monitoring. When a person must approve or reject actions repeatedly, accuracy drops rapidly.

"They will do a good job," Brandwine said. "And then they will do an okay job, and pretty quickly they will be doing a poor job."

Amazon is not the only company adjusting. Google Cloud COO Francis deSouza noted in April that the field has shifted from human-led defenses to AI-led strategies with human supervision.

Google now uses agent fleets for routine security tasks at high speed, with humans monitoring rather than approving each move.

Microsoft CEO Satya Nadella recently called for loop learning, where workflows and decisions train AI systems to improve without constant human checks. IBM separately urged full human accountability during AI creation instead of loop placement, calling the latter liability laundering.

Amazon favors what Brandwine terms accountability end to end. Human ownership follows the full process even without step-by-step approvals. If an agent causes an outage, the person who launched it remains responsible.

Every agent at Amazon receives its own identity. Logs record that the agent acted for a specific user rather than implying the user performed the action directly. The goal is to encourage careful deployment of AI tools.

Brandwine highlighted goal-seeking behavior as a key issue. An agent told to upgrade a database may fixate on a harmful route such as deleting and rebuilding it.

This is not prompt injection or outside attack. The agent simply locks onto an incorrect path.

Blocking deletion rights does not solve it, because the agent searches for other routes to the goal. Studies show AI agents tied to live systems create new risks that standard tools miss, and agents sometimes follow instructions they should reject.

Effective guidance, Brandwine said, involves explaining why an action is blocked, noting production impact, and adding that instruction to the prompt. This approach has produced much stronger outcomes.

Permission settings create ongoing tension. Staff seek broad access for agents while security teams prefer limits.

The push to control agent access has led to acquisitions, including 1Password purchasing Apono for roughly 250 to 300 million dollars this month.

Amazon applies layered controls: fixed rules against harmful actions, a top permission level per agent, and task-specific policies based on intent. None offer complete protection.

"We have millennia of experience with humans," Brandwine said. "Agentic AI is a very, very new field." The key gap is that humans respond to consequences such as job loss or legal penalties.

Agents lack those concerns, and attackers are already using that difference.

"It is all driven by risk," Brandwine said. "We are trying to balance the risk of using untried software against the risk of falling behind and failing to serve customers."