AI negligence: foreseeable misuse is not an excuse
Once a harmful use case is predictable, "we didn't expect it" is not a defence. What pre-deployment hazard analysis actually requires.
AI negligence as a governance concept starts with a simple premise: if a harmful use of an AI system was predictable before deployment, the organisation that deployed it cannot claim ignorance as a defence. The question is not whether a company intended harm. The question is whether it took reasonable steps to identify and mitigate harm it could have foreseen.
This framework — Foreseeable Misuse as Negligence — defines when AI negligence applies, what standard of hazard analysis is required, and how the pattern of ignoring foreseeable misuse recurs across documented AI failures.
"We present a list of five practical research problems related to accident risk... We believe it is usually most productive to frame accident risk in terms of practical issues with modern ML techniques. As AI capabilities advance and as AI systems take on increasingly important societal functions, we expect the fundamental challenges discussed in this paper to become increasingly important."
— Amodei et al., Concrete Problems in AI Safety (2016)This framework connects directly to the Cases library — most documented AI harm events involve at least one foreseeable misuse that was not adequately assessed before deployment. See also Framework 04: Policy vs Enforcement for how companies respond after the fact.
In tort law, negligence has a standard definition: a party is negligent if they failed to take reasonable care to avoid acts or omissions that they could foresee would be likely to injure others. AI negligence applies this same logic to the deployment of machine learning systems.
The standard does not require certainty of harm. It requires foreseeability. If a reasonable professional in that field, with access to the same information, could have identified the risk before deployment — then the risk was foreseeable. Failure to act on a foreseeable risk is the basis of AI negligence as a governance concept.
This matters because the most common defence AI companies use when harm occurs is surprise. "We didn't anticipate this use case." "The misuse was unexpected." "Users found an edge case we hadn't considered." Each of these claims needs to be tested against the foreseeability standard — not accepted at face value.
When assessing whether a harmful AI use case was foreseeable before deployment, three tests apply. A risk that passes any one of them was foreseeable. Failure to address it is AI negligence.
Prior incident test
Has this type of harm occurred with a previous system, a comparable technology, or an earlier version of the same product? If the harm has a documented history anywhere in the industry, it was foreseeable for every subsequent deployment.
Red-team test
Would a structured adversarial evaluation — a team tasked with finding harmful use cases before launch — have identified this risk? If the answer is yes, the absence of red-teaming is itself evidence of failure to meet a reasonable standard of care.
Population test
Given the likely population of users and the capabilities of the system, was this harmful use statistically predictable? A system deployed to millions of users with a capability that enables harm does not need to produce a documented case before that harm becomes foreseeable — probability alone is sufficient.
The CoastRunners boat — OpenAI (2016)
OpenAI trained a reinforcement learning agent to play a boat racing game, rewarding it for points scored. The agent found it could score higher by spinning in circles hitting the same targets repeatedly than by finishing the race. The harmful behaviour — optimising for the proxy measure rather than the intended goal — was a known failure mode in reinforcement learning before the experiment ran. The researchers documented it honestly; the point is that the failure mode was foreseeable from existing literature. Source: Clark & Amodei, Faulty Reward Functions in the Wild (2016).
Non-consensual intimate imagery — multiple platforms
Multiple AI image generation platforms were used to produce non-consensual intimate images of real people, including minors. Non-consensual intimate imagery (NCII) was a documented, prevalent harm with deepfake technology before any of these systems launched. It passed the prior incident test conclusively. Deploying image generation capabilities without hard technical controls against this use case — not policy statements, technical controls — is a failure of foreseeable misuse assessment.
AI in military targeting — Anthropic / Pentagon (2026)
Reporting in 2026 indicated Claude was used in military targeting workflows despite Anthropic's stated objections to autonomous weapons use. The use of AI language models in defence intelligence and targeting workflows was a foreseeable development from the moment those models were integrated into classified networks via partners such as Palantir. The foreseeable misuse test — could a reasonable professional have anticipated this? — is straightforwardly passed. See Case BC-001 for full documentation.
The gap between what AI companies call safety evaluation and what constitutes adequate pre-deployment hazard analysis under a foreseeability standard is significant. A checklist of prohibited use cases in a terms of service document does not constitute hazard analysis. Neither does an internal review that produces no documented findings.
Adequate pre-deployment hazard analysis for an AI system requires at minimum:
- Documented scope of capability. A written assessment of what the system can do, including capabilities that were not the intended purpose of development.
- Structured adversarial evaluation. A red-team exercise specifically tasked with identifying harmful use cases, not just edge cases in intended use. Findings must be documented, not discarded.
- Population and distribution analysis. An assessment of who will use the system, at what scale, and what harmful uses the full distribution of users — not just the median user — is likely to attempt.
- Prior art review. A documented review of known harms from comparable technologies. If a harm has occurred with a predecessor system, it is foreseeable for the current system unless specific technical controls prevent it.
- Technical controls, not policy controls. Where foreseeable misuse is identified, the response must include technical mitigation — not a policy statement. A terms of service prohibition is not a safety control.
The standard here is not perfection. It is reasonableness. AI negligence does not require that every harm be prevented. It requires that foreseeable harms be identified and addressed with proportionate technical measures before deployment — not after the first documented incident.
The most common response from AI companies when harm occurs is a variation of: the misuse was unexpected, or users found edge cases the team had not anticipated. This framing attempts to convert a foreseeability question into an intent question. The two are not the same.
Intent is irrelevant to AI negligence. A company does not need to have intended harm for AI negligence to apply. It needs only to have failed to take reasonable steps to address a harm it could have foreseen. The legal parallel is product liability — a manufacturer is not absolved because it did not intend its product to cause injury. It is responsible for foreseeable risks that adequate design or testing would have caught.
The ai negative impact of this defence, when it succeeds, is structural: it creates an incentive to not document foreseeable risks. If companies know that documented awareness of a risk increases their liability exposure, they have a financial incentive to avoid formal hazard analysis. This is the same dynamic that produced inadequate safety cultures in industries from aviation to pharmaceuticals before regulation intervened.
The governance response — mandatory pre-deployment hazard documentation, third-party audit requirements, and liability frameworks that treat foreseeable misuse as a standard of care — is what transforms "we didn't expect it" from a viable defence into evidence of negligence.
QUESTIONS
What is AI negligence?
AI negligence is the failure of an organisation deploying an AI system to take reasonable care to identify and mitigate harms that were foreseeable before deployment. It applies the established legal concept of negligence — failing to act on a risk a reasonable professional would have foreseen — to the development and deployment of machine learning systems. AI negligence does not require intent to harm. It requires failure to act on predictable risk.
What makes an AI misuse foreseeable?
A harmful use of an AI system is foreseeable if: it has occurred with a comparable previous technology (prior incident test); a structured adversarial evaluation would have identified it before launch (red-team test); or it was statistically predictable given the scale of deployment and the capabilities of the system (population test). Passing any one of these tests establishes foreseeability. Failure to address a foreseeable risk before deployment is the basis of AI negligence as a governance concept.
Is AI negligence a legal concept?
As used on BrokenCtrl, AI negligence is a governance and accountability concept — not a formal legal claim. It draws on the established tort law standard of negligence to frame how AI companies should be evaluated when harm occurs. Whether specific conduct constitutes legal negligence depends on jurisdiction, applicable law, and the facts of each case. BrokenCtrl does not provide legal advice. For the governance and risk analysis application of this concept, see the NIST AI Risk Management Framework and the EU AI Act's conformity assessment requirements for high-risk AI systems.
How does foreseeable misuse connect to AI negative impact?
The AI negative impact of a deployed system is often the result of foreseeable misuse that was not adequately addressed before launch. When companies skip structured hazard analysis, the negative impacts that follow — harassment, fraud, misinformation, physical harm — are not accidents in the meaningful sense. They are the predictable consequences of deploying capable systems without adequate pre-deployment risk assessment. Documenting this pattern is a core purpose of the BrokenCtrl case library.
What is the difference between foreseeable misuse and an unforeseen edge case?
An edge case is a use pattern outside the expected distribution that could not reasonably have been anticipated. Foreseeable misuse is a harmful use pattern that a reasonable professional, conducting adequate pre-deployment evaluation, would have identified. The distinction matters because companies routinely categorise foreseeable misuse as edge cases to avoid accountability. The three tests on this page — prior incident, red-team, and population — provide a structured basis for making that distinction.