Human-in-the-loop Is Not the Safety Net You Think It Is
Human-in-the-loop oversight for AI coding agents is a security design where humans must approve or deny an agent’s commands or changes before they run, in the hope that human judgment will reliably filter out dangerous actions and reduce autonomous AI risks by serving as a last line of defense against malicious or careless behavior.
That design promise is failing in practice. A large-scale browser-based experiment on AI agent security found that humans approved roughly one in three malicious requests even when they knew they were acting as gatekeepers. The very mechanism meant to protect us is leaking dangerous commands at a rate that would get any human-operated SOC torn apart in a post-incident review. Meanwhile, AI models themselves “may not be that good at fixing security flaws” when left alone, with autonomous patches fully fixing vulnerabilities only about a quarter of the time. So we have a double bind: humans are sloppy, and agents are unreliable. Pretending human oversight alone solves autonomous AI risks is not prudence; it is security theater.
Where Humans Miss: Credentials, Configs, and Subtle Payloads
The most alarming part of these human oversight failures is what people miss. In the experiment, one in three malicious commands slipped past human reviewers on average. These were not obscure edge cases. The most commonly overlooked were scope violations such as an AI agent requesting to cat Kubernetes configuration files or AWS credentials lists, which could “easily lead to the sensitive data they contain being exfiltrated,” and were missed about 35 percent of the time. This is credential exposure by design, not by accident. If your AI agent can read kube configs or credential stores on demand, your infrastructure is one sloppy click away from compromise.
People are better at spotting obviously destructive commands like rm -rf / or recursive chmods, which were frequently caught. But attackers do not need blatant destruction when they can quietly walk out with your secrets. Curling unknown APIs and installing typosquatted packages were also missed at rates similar to those scope violations, underscoring how thin human intuition is against subtle supply-chain or exfiltration paths. The single most frequently missed malicious command was npm run analyze, approved nearly 65 percent of the time even though it can run arbitrary scripts defined in package.json. When developers cannot be bothered to read the agent history log sitting right above the prompt, human-in-the-loop becomes a rubber stamp, not a safety buffer.
Why Human Judgment Buckles Under Agent Workflows
To understand these human oversight failures, look at the incentives and the workload. Players in the experiment had 60 seconds to review as many commands as possible, losing points for both approving malicious actions and blocking safe ones. That pressure mirrors real-world development environments, where agents fire off a stream of prompts and humans are expected to quickly triage them while trying to get work done. The result is cognitive fatigue. The experiment’s creator called manually approving every command “a draining activity that invites disaster,” noting that noisy, repetitive prompts make developers less diligent. According to one telemetry report, users approve around 93 percent of permission prompts, and “the more approvals a user sees, the less attention they pay to each.”
The broader shift in development workflows makes this worse. We have moved from reviewing single-line suggestions to handing off complex tasks and reviewing only the final diff, “letting the agent churn and iterate until then,” which has been described as a recipe for disaster. Combined with fragile AI output—where many LLM-generated patches are “Fix-Like Artifacts With Embedded Defects” and the expected value of a fully LLM-generated, non-human-reviewed patch is “a net-negative by a considerable margin”—you get the worst of both worlds. Humans grow numb, models misfire, and nobody owns the risk in the middle.
Autonomous AI Risks: FLAWED Patches and False Confidence
Security leaders often comfort themselves with the mantra that human oversight will catch what AI misses. The evidence points the other way. In one study of 6,080 AI-generated patches across six CVEs using two frontier reasoning models, only 26.0 percent fully resolved the vulnerabilities without changing application behavior. Another 20.1 percent fixed issues but altered behavior, 49.3 percent failed to close at least one exploit path, and a non-trivial fraction introduced new security issues. The authors coined the acronym FLAWED—Fix-Like Artifacts With Embedded Defects—to describe these patches, concluding that the expected value of a non-human-reviewed patch is negative. In other words, blindly trusting autonomous agents to remediate security problems is itself a security problem.
When you combine FLAWED patches with inattentive human reviewers, the system becomes dangerous by construction. Developers are encouraged to rely on human-in-the-loop frameworks as a cure-all, but one researcher bluntly observed that tooling must make systems safer “than pointing to HITL as a valid solution.” Yet current frameworks often encourage default-approve behavior because stopping a multi-hour agent flow five minutes in is painful. That is how you end up with people flipping to “dangerously skip permissions” modes or reflexively hitting approve. The false sense of security is more damaging than acknowledging that human supervision, as currently implemented, cannot keep up with autonomous AI risks.
What Enterprises Must Do Now: Beyond Human-In-The-Loop
If you are deploying autonomous AI agents in your organization, relying on human oversight alone is irresponsible. You need structural controls that assume human reviewers will miss around a third of dangerous actions and that credential exposure is inevitable if secrets are within reach. Stronger AI agent security starts with isolation: run agents in sandboxes or devcontainers in the cloud, and treat their environment as potentially hostile. Limit filesystem and network scope so the agent cannot access Kubernetes configs, AWS credential stores, or production secrets even if a human clicks approve.
Next, rework permission models. It “becomes clear we need to pay more attention to the permission model of these agents, and devs need to be more aware of the trade-offs of them.” Introduce policy-based approvals where possible, and use model-based classifiers to pre-filter commands before they reach humans. One provider built an “auto mode” that delegates some decisions to a classifier specifically to reduce approval fatigue. Finally, instrument your pipelines: log every agent command, tie it to approvals, and continuously test patches—using tools like FLAWED-style harnesses—to detect Fix-Like Artifacts With Embedded Defects. The conclusion is uncomfortable but necessary: autonomous AI risks demand automated defenses. Human-in-the-loop is a component, not a shield.
