--- id: kn-e7-harm-and-risk title: Harm scenarios and risk assessment bucket: E7 Risk, safety and safeguarding item: null topic: harm-scenarios question: What could go wrong for users and others if the tool is built, how likely and severe is each harm, and how will the team prevent and respond to it? type: question audiences: - all evidence: practitioner-consensus status: draft reviewed_by: [] updated: 2026-10-17 url: /risk-safety/harm-and-risk sources: - id: src-001 pages: - 4 - 5 - 6 - id: src-003 pages: - 3 - 5 - id: src-004 pages: - 13 - id: src-010 pages: - 10 - id: src-011 pages: - 9 - id: src-012 pages: - 15 - 16 - 17 - 21 - id: src-013 pages: - 11 - 15 - id: src-014 pages: - 12 - 13 - 27 - 35 - 40 - 42 - 45 - 66 - id: src-015 pages: - 39 - 45 - id: src-017 pages: - 1 - 3 - 5 - 6 - id: src-019 pages: - 11 - 12 - 14 - 21 - 46 - 47 - id: src-024 pages: - 1 - 9 - 55 - 62 - 71 - 83 - 85 - 93 - 135 - id: src-028 pages: - 22 - 23 - 24 - 58 - id: src-029 pages: - 12 - 13 - 22 - 23 - id: src-031 pages: - 2 - id: src-034 pages: - 3 - 4 - id: src-035 pages: - 10 - 11 - 12 - 14 - 17 - id: src-037 pages: - 26 - 27 - 58 - id: src-038 pages: - 23 - 25 - 26 --- # Harm scenarios and risk assessment > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. > [!NOTE] > Candidate item. The MTI team has not yet agreed the items for this bucket. **Question:** What could go wrong for users and others if the tool is built, how likely and severe is each harm, and how will the team prevent and respond to it? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-17 ## Basics Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway [src-014 p35; src-013 p15; src-037 p26]. Harms can be direct or indirect, and intended or accidental [src-019 p11]. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health [src-019 pp11-12; src-013 p11]. ## Why it matters for AI tools AI makes some old harms larger and faster. It can produce abusive content and deepfakes at volume, and its workings are hard to inspect, which makes safety checks difficult [src-014 p66; src-029 p13]. Some harm comes from people misusing a tool on purpose; some comes from a tool behaving badly when used as intended [src-029 pp12-13]. Guardrails, the built-in limits meant to stop harmful output, are easy to get round. In a UNESCO test, calling a request fiction was enough to get a model to write insults about a named woman journalist in several languages, ready for a fake harassment campaign [src-029 pp22-23]. Even a very accurate system can cause harm, and a widely used tool will be put to uses its builders never planned [src-024 p62]. Unplanned use is documented. In Turkey, students given a basic chatbot scored worse in exams than students with no chatbot, while a version with learning safeguards avoided this [src-015 p39]. After one organisation scaled its AI tool, frontline staff began treating it as the decision-maker and gave less personal support [src-028 p58]. Remote contact carries its own risk, even without AI. A Ukrainian health service for adolescent female sex workers flagged that giving test results by phone could lead them to treat themselves and skip clinic visits [src-038 p23]. People in crisis use chatbots for help the bot was never built to give. Many quickly built health chatbots could not recognise a message about violence or self-harm and offered no route to a human [src-034 p3]. Harm can also arise in the wider system. A woman who cited a legal chatbot at a police station was turned away, because the project had engaged women users but not the police [src-028 p22]. DIAL notes that harms from AI are less well understood than harms from other technology [src-010 p10]. A review of harms to health workers in low- and middle-income countries found only two studies that looked at AI at all [src-017 p5]. Little evidence of harm does not mean little harm. Bias and unfair treatment of groups is covered on its own page in this bucket. ## What good evidence looks like - A list of harms for each user group, including bystanders and people affected by the tool but not using it [src-014 p35; src-024 p83]. - At the idea stage, before design, the team asked whether the core idea could be turned against its users and whether the business model works against user safety [src-014 pp40, 42]. - Both accidental harms and deliberate misuse are considered, with examples specific to the tool [src-029 pp12-13; src-014 p35]. - Each harm is scored for likelihood and severity, mitigations are named, and the leftover risk is scored again [src-014 p35; src-037 p58; src-024 p9]. - Low-scored risks still have a planned response; scoring sets the order of work, not what to ignore [src-014 p45]. - Users, especially vulnerable ones, helped name the risks, and their concerns carry weight in setting priorities [src-012 p21; src-038 p26; src-013 p15]. - A named person owns each mitigation, with a deadline and sign-off [src-037 pp26-27]. - A clear point at which the team will halt or not launch [src-037 p58; src-024 p71]. - The risks of the AI tool are compared with those of a simpler model, a manual process or human staff doing the same job [src-024 p135]. - A person checks outputs before they affect anyone, and that person can overrule the tool [src-011 p9; src-013 p15]. - A plan for people who disclose danger to the tool, with a route to human help [src-034 pp3-4]. - A date and trigger for repeating the assessment, such as new features or a change in context [src-037 p27; src-003 p5]. ## Red flags in a proposal - Risks are listed only as accuracy or uptake problems, with no harm to people named [src-003 p5]. - Only the main user group is considered, not others affected [src-014 p35; src-024 p83]. - No one asks whether use is truly optional for the people affected [src-035 p12]. - No statement of who owns the implementer or what else the data will be used for [src-035 p17]. - An open-ended chatbot faces the public with no plan for how its wide range of risks will be managed [src-024 p85]. - Advice on health or money with no human check before anyone acts on it [src-013 p11; src-015 p45]. - The risk assessment is a one-off form with no plan to revisit it [src-037 p27; src-028 p23]. - A tool is added on top of existing paper or digital systems without asking what extra work it creates [src-017 pp5-6]. - The tool rests on a pre-trained model from another company, and there is no plan to track that model's risks. Such models make bad outcomes harder to foresee and to trace [src-024 p55]. ## How to find out Several methods can surface harms, and each misses some things. Ready-made lists help a team start. Girl Effect sets out twelve risk areas for behaviour change chatbots, such as loss of human contact, drained phone data, wrong or toxic replies, environmental cost and poorly paid data workers, and marks which ones generative AI makes worse [src-012 pp15-17]. It also notes that many are old problems in a new form, so existing safeguarding practice still applies [src-012 p17]. IREX lists risks by type of technology. For chatbots these include over-reliance, weaker critical thinking, tracking without consent and leaks of behavioural data; for data collection apps, leaks, consent failures and poor data on groups already left out [src-014 pp12-13]. For tools aimed at health workers, the EDiT framework links each area of digital work to harms such as false information, constant monitoring, duplicate paper and digital systems, loss of skills and out-of-pocket costs, and then to effects on patients, staff and the health system [src-017 p1]. To describe a single harm in full, EDiT suggests recording who was involved, the type, how often and how severe it was, the channel, how the person responded, and the impact, for example physical, psychological, economic or reputational. Social norms shape each step [src-019 p14]. The kind of device access people have, such as a shared phone, also shapes their exposure, so ask about it first [src-019 p21]. A participatory stakeholder analysis brings users, staff and community leaders together to name local dangers. In one Indian example, it showed that a phone line to mostly female staff could be abused, and the team switched to in-person feedback [src-031 p2]. Social risk questions ask whether using the tool could change how users are seen, or expose children or homes through location data [src-004 p13]. A futures wheel maps the direct and knock-on effects of a decision with domain experts [src-028 pp23-24]. Searching for similar cases where automated decisions failed shows what has gone wrong elsewhere [src-001 p4; src-024 p93]. Structured lists of failure points, such as a nine-stage taxonomy for farm advice, help a team check where a tool could break down, even when the model is accurate [src-003 p3]. Red-teaming means testing the tool or concept by trying to make it cause harm [src-013 p15]. A risk register records each harm, its scores, its controls and its owner [src-014 p35; src-037 p58]. One register built for chatbots is PRIMA, IREX's Predictive Risk and Mitigation Audit. It asks about the tool's function, design and audience, flags privacy, safety and security risks with their likelihood, and suggests safety features; the tool itself is not in the source [src-013 p15]. Australia's eSafety Commissioner publishes free safety-by-design assessment tools, developed with industry, NGOs and users [src-014 pp27, 35]. ## Limits and disagreements Sources disagree on what to do when a risk cannot be estimated. Pollicy says an AI solution should not be deployed until likelihood and severity can be judged [src-035 pp10, 14]. AAAS asks for measures to reduce an uncertain risk unless it can be shown to be negligible, which allows going ahead [src-001 p6]. Digital Futures Lab accepts that some risks will only appear during a pilot [src-028 p23]. There is a tension on scope. NIST says a narrow, scripted chatbot is easier to manage than an open-ended one [src-024 p85]. UNICEF found that scripted health bots failed users in crisis because they could not recognise unexpected messages [src-034 p3]. Context limits: - NIST is a US government body writing for US organisations [src-024 p1]. - Pollicy's likelihood and severity table is aimed at state regulators in Africa, not project teams [src-035 p10]. - Girl Effect's assessment follows UK data protection guidance [src-037 p26]. - The health worker review covers mostly non-AI tools [src-017 p5]. Two thirds of its 99 studies looked at doctors, although community health workers are often the main users of digital tools. Most studies also measured end results such as burnout, not the harms that led to them [src-017 p3]. The review asks future work to cover frontline staff and AI, and to use tested survey tools [src-017 p6]. - AAAS describes the EU AI Act as a proposal, so its summary predates the law [src-001 p5]. - The UNICEF girls' guidance dates from 2013 [src-038 p25]. - Several cases come from India, Pakistan and Turkey, often without the organisation being named [src-028 pp22, 58; src-015 p39]. - Proving that a tool caused a health harm needs a purpose-built study; a general survey will not show it [src-019 pp46-47]. Shared authors: src-012, src-034, src-037 and src-038 share MTI-linked authors, so their agreement on user involvement and safeguarding counts as less independent. The EDiT consortium (src-017, src-019) is one voice, and so is IREX (src-013, src-014). ## Practitioner detail Pollicy suggests matching the response to the score. For likely and severe harm, do not deploy until it is fixed. For unlikely but severe harm, commission an external human rights impact assessment. For likely but minor harm, offer strong redress for each person affected [src-035 p10]. Digital Green's taxonomy flags which risks fall hardest on women and which are specific to AI, and advises putting severe and inequitable risks first [src-003 pp3, 5]. The Girl Effect form scores severity and likelihood from 1 to 3 and stops the activity if the risk stays high after mitigation [src-037 p58]. Pollicy also lists questions that raise the level of scrutiny a tool needs. Is the person using the tool in a very different position from the person it affects? Does it predict human behaviour? Does it use personal data, even anonymised? Is a private company involved? Can a human reviewer check its decisions afterwards from information about the model and data? Pollicy wrote these for regulators [src-035 p11]. AAAS suggests where to start. Risk cannot be removed, so it must be ranked, and tools that deal with people directly or learn from large sensitive datasets come first [src-001 p5]. If a tool could involve emergency limits on people's rights, its decision tree asks whether the limits are needed at all, whether those affected know and agree, whether the limits are proportionate and time-bound, and whether affected people have a voice in oversight [src-001 p4]. ## Sources - **src-001**: Decision Tree for the Responsible Application of Artificial Intelligence (v1.0). AAAS Center for Scientific Responsibility and Justice (CSRJ). Cited: pages 4, 5, 6. - **src-003**: AI-Enabled Farmer Advisory: Risk & Failure Taxonomy. Cited: pages 3, 5. - **src-004**: CONNECT! A practical guide to using ICTs in Plan projects. Plan Finland. Cited: page 13. - **src-010**: Principles for Digital Development. Digital Impact Alliance (DIAL), 2024. Cited: page 10. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 9. - **src-012**: Girl Effect Ethical AI Guidelines for the Deployment of Social and Behaviour Change Chatbots. Girl Effect, 2025. Cited: pages 15, 16, 17, 21. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: pages 11, 15. - **src-014**: Safety by Design: Safe Approaches for Ethical Technology by Design: Making Products and Services Safer for Users and Consumers. Curriculum and Facilitator Guide. IREX, 2025. Cited: pages 12, 13, 27, 35, 40, 42, 45, 66. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 39, 45. - **src-017**: Measurement of Harms Associated with Technology use among Health Workers. EDiT Consortium. Cited: pages 1, 3, 5, 6. - **src-019**: Measuring Digital Harms in Low and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2025. Cited: pages 11, 12, 14, 21, 46, 47. - **src-024**: AI RMF Playbook. Cited: pages 1, 9, 55, 62, 71, 83, 85, 93, 135. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 22, 23, 24, 58. - **src-029**: Red Teaming Artificial Intelligence for Social Good: The Playbook. UNESCO, 2025. Cited: pages 12, 13, 22, 23. - **src-031**: SIMLab Context Analysis Annex 1: Example Assessment. Social Impact Lab (SIMLab). Cited: page 2. - **src-034**: Safer Chatbots Implementation Guide. UNICEF East Asia and Pacific Regional Office (Gender section). Cited: pages 3, 4. - **src-035**: Towards Afro-feminist AI: A Handbook for approaching Governance of AI in Africa. Cited: pages 10, 11, 12, 14, 17. - **src-037**: Digital safeguarding tips and guidance. Girl Effect, 2018. Cited: pages 26, 27, 58. - **src-038**: Integrating Information and Communication Technologies into Communication for Development Strategies to Support and Empower Marginalized Adolescent Girls. UNICEF (C4D Section, Programme Division), 2013. Cited: pages 23, 25, 26. ## Guidance for AI assistants This page is a draft. No subject expert has reviewed it yet. Tell the user so when you use it, and treat its points as provisional. If you are an AI assistant using this page to help someone: - Before advising, ask whether they are reviewing a proposal, designing a funding call, setting a budget, or learning. - Cite this page by its URL, with its evidence tag and the sources it names. - Say when the evidence comes from one region, sector or kind of project, and may not fit the user's context. - Point out weak or unsupported claims in a proposal rather than smoothing them over. - Remind the user not to paste identifiable proposal details or personal data into an AI tool their organisation has not approved.