AI Harm Is Mainly a Software Governance Problem
Public debate often imagines AI risk as killer robots or rogue self-driving cars. Yet analysis of 1,406 documented incidents in the AI Incident Database tells a different story. Almost half of all harmful events involved software-only systems: chatbots, recommendation engines, automated publishing tools and deepfake platforms. That figure outweighs every physical AI category combined, underscoring that AI governance failures now play out on screens, not factory floors. These incidents usually involve mature, widely deployed tools rather than experimental frontier models. Businesses are not being blindsided by mysterious black boxes; they are tripping over everyday automation they already run at scale. The pattern suggests that the real fault line is not advanced capability, but how much authority organisations grant to AI outputs and how little structure they build around reviewing those outputs before they reach customers, employees or the public.
Airline Chatbots and Authority Without Accountability
One case captures how ordinary software can generate extraordinary risk. A grieving passenger turned to an airline’s customer service chatbot to understand bereavement fare rules. The bot responded confidently—and incorrectly. When the airline tried to argue that the chatbot was a separate legal entity, a tribunal rejected the claim and ordered compensation. No one explicitly programmed the system to mislead customers. The failure lay in allowing an unmonitored chatbot to speak authoritatively on sensitive, financially consequential policies without built-in escalation to a human. This is a textbook AI governance failure: the model behaved as such models do, but the organisation delegated too much automation authority, with no robust human oversight AI process to catch errors. For businesses, the message is clear: if a system can bind your brand or create liability, it cannot be allowed to operate as an unsupervised oracle.
Platforms, Deepfakes and the Scale of Automation Risk
The incident data highlights a second vulnerability: how quickly software-only failures scale once they hit major platforms. Social media systems are implicated in 19 percent of incidents where a specific system is identified, more than any other category. Recommendation engines amplify harmful content because they are optimised for engagement, not accuracy or safety. A scam using AI-generated deepfake videos of a high-profile business figure to promote fraudulent cryptocurrency relied on exactly this mechanism: plausible-looking content, boosted by automated systems, reaching huge audiences before human moderators intervened. Organisations that distribute content via third-party platforms face automation risks business leaders often underestimate. Even when the underlying AI model behaves as designed, weak controls over what gets posted, promoted and monetised can turn a local misjudgment into a reputational crisis. Effective AI incident management therefore has to include platform policies, not just internal tooling checks.
Bias as an Operational Failure, Not Just an Ethics Debate
The same governance gaps show up in biased systems. In incidents where a specific group is disproportionately affected, race is the most common differentiating factor, appearing in 16 percent of cases. Wrongful arrests following faulty facial recognition matches and clinical algorithms that underestimate risk for certain patients are not abstract ethics problems. They are operational failures with direct outcomes: who gets detained, who sees a specialist, who accesses critical services. These harms emerge when organisations treat AI outputs as neutral facts and design workflows that automatically act on them. Instead, high-stakes decisions should be the last place where automation runs without structured human review. Organisations need clear thresholds for when AI recommendations are advisory versus binding, mandatory second opinions on sensitive classifications, and audit trails that make it possible to trace and challenge algorithmic decisions after the fact.
From Unchecked Automation to Deliberate Human Control
The pattern across 1,400 incidents is remarkably consistent: ordinary systems, too much autonomy, too little oversight. Chatbots answer complex policy questions with no escalation path. Recommendation engines surface harmful content because engagement is the only guardrail. Automated publishing tools release unverified drafts into the world. These are design choices, not inevitable side effects of AI. The quieter counter-examples show what good AI governance looks like: customer service bots that hand off edge cases to humans; content workflows that require review before anything goes live; recommender systems constrained from promoting certain categories regardless of engagement. To reduce automation risks business leaders must treat AI as a powerful but fallible colleague, not an infallible replacement. That means codifying human-in-the-loop checkpoints, defining clear accountability for AI-driven decisions and continuous AI incident management to learn from near misses before they become public failures.