The uncomfortable truth: people are bad at policing AI
AI security vulnerabilities increasingly stem from human decisions, because people struggle to reliably spot malicious AI-generated requests, phishing messages, and impersonation attempts even when they know such attacks exist and believe they are being careful. That gap between confidence and performance turns human oversight into a fragile control, not a dependable safety net. In other words, the more you lean on people to police AI, the more you invite subtle failures that attackers can automate and scale. A browser-based test of an AI coding agent’s permission prompts found that humans approved roughly one in three malicious commands, including scope violations such as requests to read Kubernetes configuration files or AWS credential lists that could lead to data exfiltration. That is not a rounding error; it is systemic human oversight failure under time pressure and noise. If you are relying on “review every prompt” as your main defense, you are architecting around wishful thinking, not real human behavior.
Why human-in-the-loop breaks down with coding agents
The coding-agent game shows why human oversight fails in practice: fatigue, lack of context, and skewed risk perception. Players had to approve or deny as many simulated commands as possible in 60 seconds, with missed malicious requests and blocked safe commands both penalizing their score. The data from more than 40,000 runs revealed that one in three dangerous requests slipped through human gatekeepers, with scope violations missed around 35 percent of the time. Destructive commands like wiping the root directory were easier to catch; subtle credential theft was not. Real telemetry points in the same direction: users of popular coding agents approve about 93 percent of permission prompts, which means almost everything gets a green light in day-to-day workflows. Under that volume, “approve every step” becomes noise. As one analysis put it, “The high amount of noise introduces fatigue, and developers don’t always have the context of what has changed to quickly determine the risk”.
If you keep piling prompts on developers, many will choose the equivalent of “dangerously skip permissions” so they do not stall long-running agent flows. That is rational behavior from their point of view and a disaster from a security perspective. This is why defenders are experimenting with model-based classifiers to take over some approvals: one major vendor built an auto-approval mode to reduce fatigue, delegating certain decisions to another model, explicitly describing it as a layer of defense-in-depth inside a sandbox rather than a replacement for sandboxing. That design choice matters; trusting classifiers without isolation would repeat the same mistake of overestimating oversight, just with silicon instead of humans.
AI-powered social engineering: voice clones and phishing texts
The most worrying shift is from attacking systems to manipulating people. Recent campaigns against large investment firms used AI-generated voice clones to impersonate senior executives and colleagues during routine phone calls, aiming to trick staff into giving up login credentials or other internal access. These calls focused on deception rather than malware or network exploits, trying to persuade employees to bypass security controls rather than break through them. That is a targeted social engineering threat amplified by AI, not a traditional technical hack. Text-based attacks are evolving in the same direction. In a pilot study, 25 volunteers received twelve spear phishing messages tailored to their job, hobbies, and social media posts; half were written by GPT-4, half by trained students. Participants were asked to sort them by click likelihood and then guess which were AI-generated. Across 300 judgments, they identified AI phishing texts correctly only 52 percent of the time, barely above chance.
The study’s most important finding is not whether GPT-4 beats humans at writing phishing—it is how easily scalable AI attacks match curated human phishing. A single short prompt, filled from a survey and run once per person, produced AI messages that landed above participants’ “I would click” line 28 percent of the time, compared with 21.3 percent for student-crafted texts that had been filtered by a review team. Job-themed messages were the most effective: they crossed the click threshold 38 percent of the time, far ahead of hobby-based (19 percent) and social-media-based (17 percent) lures. That tells you where AI will aim: work-related trust channels that feel routine and urgent, the very messages traditional awareness training teaches people to act on quickly.
Training is not enough when attackers scale persuasion
Security teams have long treated awareness training as the main answer to social engineering threats. In the AI era, that assumption breaks. The hedge fund voice-clone campaign reached multiple firms almost simultaneously, turning what used to be a one-organisation-at-a-time tactic into a scalable operation. Text attacks show the same pattern: GPT-4 messages, generated from minimal inputs, performed in the same range as hand-crafted phishing refined by experts. Gut feeling about “robotic tone” did nothing; participants guessed AI versus human text at coin-flip levels. The Brigham Young study also highlights where AI still falters: getting personal details right. Wrong names or inaccurate references (“there’s no Mike at work”) gave recipients concrete reasons to be suspicious. But attackers can iterate prompts cheaply and use scraped data to improve their hit rate. Human error here is measured not by whether employees can spot AI, but by what they say they would click on—a proxy heavily used in phishing research that still underestimates real-world failure because it does not involve live consequences.
Relying on staff to sense AI-generated content is therefore misguided. The advice from one study was blunt: “Check the sender, the channel, the link, and the request against what you would expect to receive. Do not try to decide whether it sounds like a robot”. The lesson from recent campaigns is the same: AI supercharges social engineering threats by automating trust exploitation, not by sounding robotic. Defenders who keep investing in thematic training—more slides about suspicious links, more “think before you click” slogans—without changing their technical and process controls will keep losing the same battle. Awareness helps, but it cannot carry the full weight of defense when attackers can flood every inbox and phone line with plausible, personalised prompts at machine scale.
Practical defenses: design for failure, not perfect judgment
If human oversight is unreliable, the answer is not to give up—it is to redesign systems assuming people will miss a meaningful fraction of attacks. As one practitioner put it, “It’s a whole new world with a new set of attack vectors… it’s best to remain aware of the risks and know how to reduce them”. On the AI coding side, that means isolating agents. Wauters suggests running AI coding models in sandboxes or devcontainers, ideally in the cloud, and adding hooks that enforce contextual checks around potentially malicious actions before any automatic approval. In this design, even when a user clicks “approve” on a bad request, the sandbox limits blast radius and instrumentation flags suspicious behavior. Model-based approval systems can help with fatigue, but they must be treated like another fallible control. One vendor’s auto mode, which delegates some command approvals to a classifier based on the model’s own understanding of risk, is described explicitly as “one layer of defense-in-depth inside a sandbox, not a substitute for one”.
On the social engineering front, stop expecting staff to recognise AI content and start enforcing channel, sender, and request checks by design. For high-risk actions—changing payment details, granting new access, moving sensitive data—require out-of-band verification through known channels and multi-party approval. Assume voice calls can be cloned and SMS or email can be spoofed. Build playbooks that tell employees to treat any request for credentials or urgent action over informal channels as suspect unless validated. You also need detection that does not rely on human gut feel. In the phishing study, software analysis highlighted patterns people missed. Similar tooling can flag anomalous communication behaviour, odd timing, or content that departs from historical norms. As one security leader warned about AI-enhanced campaigns, “We will see more and more of this attackers leveraging powerful AI models, need to up the defense too”. The organisations that stay safe will be those that assume human oversight failures, contain their impact, and surround people with guardrails instead of blaming them after the fact.
