Harm scenarios and risk assessment

Basics

Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway src-014 p35; src-013 p15; src-037 p26. Harms can be direct or indirect, and intended or accidental src-019 p11. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health src-019 pp11-12; src-013 p11.

Why it matters for AI tools

AI makes some old harms larger and faster. It can produce abusive content and deepfakes at volume, and its workings are hard to inspect, which makes safety checks difficult src-014 p66; src-029 p13. Some harm comes from people misusing a tool on purpose; some comes from a tool behaving badly when used as intended src-029 pp12-13. Guardrails, the built-in limits meant to stop harmful output, are easy to get round. In a UNESCO test, calling a request fiction was enough to get a model to write insults about a named woman journalist in several languages, ready for a fake harassment campaign src-029 pp22-23. Even a very accurate system can cause harm, and a widely used tool will be put to uses its builders never planned src-024 p62.

Unplanned use is documented. In Turkey, students given a basic chatbot scored worse in exams than students with no chatbot, while a version with learning safeguards avoided this src-015 p39. After one organisation scaled its AI tool, frontline staff began treating it as the decision-maker and gave less personal support src-028 p58. Remote contact carries its own risk, even without AI. A Ukrainian health service for adolescent female sex workers flagged that giving test results by phone could lead them to treat themselves and skip clinic visits src-038 p23.

People in crisis use chatbots for help the bot was never built to give. Many quickly built health chatbots could not recognise a message about violence or self-harm and offered no route to a human src-034 p3. Harm can also arise in the wider system. A woman who cited a legal chatbot at a police station was turned away, because the project had engaged women users but not the police src-028 p22.

DIAL notes that harms from AI are less well understood than harms from other technology src-010 p10. A review of harms to health workers in low- and middle-income countries found only two studies that looked at AI at all src-017 p5. Little evidence of harm does not mean little harm.

Bias and unfair treatment of groups is covered on its own page in this bucket.

What good evidence looks like

• A list of harms for each user group, including bystanders and people affected by the tool but not using it src-014 p35; src-024 p83.
• At the idea stage, before design, the team asked whether the core idea could be turned against its users and whether the business model works against user safety src-014 pp40, 42.
• Both accidental harms and deliberate misuse are considered, with examples specific to the tool src-029 pp12-13; src-014 p35.
• Each harm is scored for likelihood and severity, mitigations are named, and the leftover risk is scored again src-014 p35; src-037 p58; src-024 p9.
• Low-scored risks still have a planned response; scoring sets the order of work, not what to ignore src-014 p45.
• Users, especially vulnerable ones, helped name the risks, and their concerns carry weight in setting priorities src-012 p21; src-038 p26; src-013 p15.
• A named person owns each mitigation, with a deadline and sign-off src-037 pp26-27.
• A clear point at which the team will halt or not launch src-037 p58; src-024 p71.
• The risks of the AI tool are compared with those of a simpler model, a manual process or human staff doing the same job src-024 p135.
• A person checks outputs before they affect anyone, and that person can overrule the tool src-011 p9; src-013 p15.
• A plan for people who disclose danger to the tool, with a route to human help src-034 pp3-4.
• A date and trigger for repeating the assessment, such as new features or a change in context src-037 p27; src-003 p5.

How to find out

Several methods can surface harms, and each misses some things.

Ready-made lists help a team start. Girl Effect sets out twelve risk areas for behaviour change chatbots, such as loss of human contact, drained phone data, wrong or toxic replies, environmental cost and poorly paid data workers, and marks which ones generative AI makes worse src-012 pp15-17. It also notes that many are old problems in a new form, so existing safeguarding practice still applies src-012 p17. IREX lists risks by type of technology. For chatbots these include over-reliance, weaker critical thinking, tracking without consent and leaks of behavioural data; for data collection apps, leaks, consent failures and poor data on groups already left out src-014 pp12-13. For tools aimed at health workers, the EDiT framework links each area of digital work to harms such as false information, constant monitoring, duplicate paper and digital systems, loss of skills and out-of-pocket costs, and then to effects on patients, staff and the health system src-017 p1.

To describe a single harm in full, EDiT suggests recording who was involved, the type, how often and how severe it was, the channel, how the person responded, and the impact, for example physical, psychological, economic or reputational. Social norms shape each step src-019 p14. The kind of device access people have, such as a shared phone, also shapes their exposure, so ask about it first src-019 p21.

A participatory stakeholder analysis brings users, staff and community leaders together to name local dangers. In one Indian example, it showed that a phone line to mostly female staff could be abused, and the team switched to in-person feedback src-031 p2. Social risk questions ask whether using the tool could change how users are seen, or expose children or homes through location data src-004 p13.

A futures wheel maps the direct and knock-on effects of a decision with domain experts src-028 pp23-24. Searching for similar cases where automated decisions failed shows what has gone wrong elsewhere src-001 p4; src-024 p93. Structured lists of failure points, such as a nine-stage taxonomy for farm advice, help a team check where a tool could break down, even when the model is accurate src-003 p3.

Red-teaming means testing the tool or concept by trying to make it cause harm src-013 p15. A risk register records each harm, its scores, its controls and its owner src-014 p35; src-037 p58. One register built for chatbots is PRIMA, IREX's Predictive Risk and Mitigation Audit. It asks about the tool's function, design and audience, flags privacy, safety and security risks with their likelihood, and suggests safety features; the tool itself is not in the source src-013 p15. Australia's eSafety Commissioner publishes free safety-by-design assessment tools, developed with industry, NGOs and users src-014 pp27, 35.

Limits and disagreements

Sources disagree on what to do when a risk cannot be estimated. Pollicy says an AI solution should not be deployed until likelihood and severity can be judged src-035 pp10, 14. AAAS asks for measures to reduce an uncertain risk unless it can be shown to be negligible, which allows going ahead src-001 p6. Digital Futures Lab accepts that some risks will only appear during a pilot src-028 p23.

There is a tension on scope. NIST says a narrow, scripted chatbot is easier to manage than an open-ended one src-024 p85. UNICEF found that scripted health bots failed users in crisis because they could not recognise unexpected messages src-034 p3.

Context limits:

• NIST is a US government body writing for US organisations src-024 p1.
• Pollicy's likelihood and severity table is aimed at state regulators in Africa, not project teams src-035 p10.
• Girl Effect's assessment follows UK data protection guidance src-037 p26.
• The health worker review covers mostly non-AI tools src-017 p5. Two thirds of its 99 studies looked at doctors, although community health workers are often the main users of digital tools. Most studies also measured end results such as burnout, not the harms that led to them src-017 p3. The review asks future work to cover frontline staff and AI, and to use tested survey tools src-017 p6.
• AAAS describes the EU AI Act as a proposal, so its summary predates the law src-001 p5.
• The UNICEF girls' guidance dates from 2013 src-038 p25.
• Several cases come from India, Pakistan and Turkey, often without the organisation being named src-028 pp22, 58; src-015 p39.
• Proving that a tool caused a health harm needs a purpose-built study; a general survey will not show it src-019 pp46-47.

Shared authors: src-012, src-034, src-037 and src-038 share MTI-linked authors, so their agreement on user involvement and safeguarding counts as less independent. The EDiT consortium (src-017, src-019) is one voice, and so is IREX (src-013, src-014).

Practitioner detail

Pollicy suggests matching the response to the score. For likely and severe harm, do not deploy until it is fixed. For unlikely but severe harm, commission an external human rights impact assessment. For likely but minor harm, offer strong redress for each person affected src-035 p10. Digital Green's taxonomy flags which risks fall hardest on women and which are specific to AI, and advises putting severe and inequitable risks first src-003 pp3, 5. The Girl Effect form scores severity and likelihood from 1 to 3 and stops the activity if the risk stays high after mitigation src-037 p58.

Pollicy also lists questions that raise the level of scrutiny a tool needs. Is the person using the tool in a very different position from the person it affects? Does it predict human behaviour? Does it use personal data, even anonymised? Is a private company involved? Can a human reviewer check its decisions afterwards from information about the model and data? Pollicy wrote these for regulators src-035 p11.

AAAS suggests where to start. Risk cannot be removed, so it must be ranked, and tools that deal with people directly or learn from large sensitive datasets come first src-001 p5. If a tool could involve emergency limits on people's rights, its decision tree asks whether the limits are needed at all, whether those affected know and agree, whether the limits are proportionate and time-bound, and whether affected people have a voice in oversight src-001 p4.