# E3 Language and literacy Questions in this bucket: - **E3.1 Preferred languages.** Which languages does the audience prefer to write in, and to speak in? - **E3.2 Code-switching.** In multilingual contexts, when, how and why does the audience mix languages? - **E3.3 Literacy.** How well can the audience read and write, and how does that affect the use case? - **E3.4 Modality.** Which ways of communicating (text, voice, voice notes, video) does the audience prefer and use comfortably? - **E3.5 Model performance.** How well do the proposed language models perform in the audience's languages, code-switching and modalities? > [!WARNING] > 2 of 2 pages in this bucket are not yet reviewed. Each says so at its top. ## Modality Page: /language-literacy/modality > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. **Question:** Which ways of communicating (text, voice, voice notes, video) does the audience prefer and use comfortably? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-23 ### Basics Find out how the intended users prefer to communicate, and which ways they can use with ease: typed text, voice calls, voice notes, images or video. Ask, too, where they get information today and through which channels, such as WhatsApp, radio or a trusted person [src-028 p23; src-027 pp34-35]. The answers differ between countries and within them, by town and countryside, gender, class and ethnic group [src-006 p14]. Voice channels work on basic phones and suit people who cannot read or are uneasy with text [src-038 p17]. Some people prefer to get information from a person rather than a tool [src-028 p23]. ### Why it matters for AI tools Most language models have been built for typed text in widely written languages. Speech models now make voice-first tools more feasible, but many low-resource languages are mainly spoken and remain poorly covered [src-015 p38]. A text chatbot can therefore shut out people who read little or speak a language rarely written down. The Digital Futures Lab playbook advises voice-based navigation, with spoken questions and audio answers, for users with low literacy or physical disabilities. Even for literate users, it suggests mixing text with images, voice notes or video [src-028 p47]. Text can also be hard for reasons other than literacy, such as poor eyesight [src-004 p13]. And some people trust in-person contact far more than any phone channel: Plan staff in Senegal found that phones saved time, but local culture responded much more to physical presence [src-039 p85]. Users may also use a service differently from its design. When Plan Benin launched an SMS line for reporting violence against children, many people phoned or asked for a call back instead of texting. The team thought this might reflect literacy, habit from other helplines, or an expectation of a live person [src-039 p87]. A tool built around one mode may miss how people actually reach out. Modality also affects cost and device needs. Many AI tools run as smartphone apps; SMS or voice chatbots reduce that need, though SMS is costly in some countries [src-015 p37]. ### What good evidence looks like - Which modes users use now and are comfortable with, for each group (women and men, girls and boys, readers and non-readers) [src-039 pp37-38; src-038 p24]. - Where users get information today, and how an AI tool would fit with or disrupt those sources [src-028 p23]. - Whether the spoken word carries more weight than written material for the users [src-039 p36]. - Named apps or channels, if the project must choose one. In cognitive testing, many people did not tell SMS from WhatsApp messages, or ordinary calls from internet calls [src-020 pp28, 30-31]. - Whether users' phones have accessibility features such as text-to-speech or voice navigation [src-018 p18]. - A sample of real user questions in the modes users would choose. A PATH project with Nigerian community health workers gathered questions as text, voice, images and short videos [src-008 p11]. - A plan for more than one mode where users differ, for example voice and text options [src-031 p8; src-038 p24]. ### Red flags in a proposal - A text-only tool for users with low literacy or a mainly spoken language [src-038 p17; src-015 p38]. - The channel is chosen because it suits the developer, not because users already use it [src-028 p23; src-006 p38]. - No evidence on whether users would rather talk to a person [src-028 p23]. - Survey data on "messaging" is taken as evidence that users use a specific app [src-020 pp28, 30]. - A single channel for everyone, with no thought for differences by gender, place or literacy [src-006 p14; src-039 pp37-38]. ### How to find out Participatory tools such as a communication matrix, where each group lists the media they use for spoken, written, image and number-based communication, filled in separately by women, men, girls and boys [src-039 pp37-38]. A facilitated workshop where users list how they get and share information for each kind of need, such as emergencies, health or confidential reports [src-031 p2]. Communication resource mapping and Venn diagrams to show preferred channels and how information flows, including informal channels [src-027 pp34-35]. Survey questions on calls, voice messages and text messages, with app names where the choice of app matters [src-020 pp28, 30-31]. Collecting real questions from users in their own words and modes [src-008 p11]. Watching how people use a pilot service, since use can differ from plans [src-039 p87]. ### Limits and disagreements Most sources are about communication channels (radio, SMS, voice lines) rather than the modes of an AI tool. The SDC guide is from 2016 [src-006 p24], the World Bank guide from 2009 [src-027 p34], and the UNICEF paper from 2013 [src-038 p17]. Only three sources discuss AI [src-015 pp37-38; src-028 pp23, 47; src-008 p11]. None compares how users respond to voice and text versions of the same AI tool. Radio still matters in many places. The SDC guide notes that in remote rural areas radio is often the only medium and the main source of information [src-006 p24]. A Level 0 study may find that the best route is not a direct AI tool at all. Voice is not always safer. Girl Effect's safeguarding guide notes that for sensitive topics video may carry more risk than audio, and text more than audio [src-037 p18]. Teams should weigh safety as well as preference. ### Practitioner detail Informal channels often matter more than formal ones. In an Eastern Africa health project, villagers took up a new practice only once local healers, rather than city doctors, carried the message [src-027 pp34-35]. The EDiT skills guide allows voice search as well as typed search in its questions, for people with low literacy [src-020 p20]. Survey questions on voice messages and internet calls should not name a single app unless the project is choosing which app to build on [src-020 pp30-31]. ### Sources - **src-038**: Integrating Information and Communication Technologies into Communication for Development Strategies to Support and Empower Marginalized Adolescent Girls. UNICEF (C4D Section, Programme Division), 2013. Cited: pages 17, 24. - **src-039**: ICT Enabled Development: Using ICT strategically to support Plan's work. Plan Finland. Cited: pages 36, 37, 38, 85, 87. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 37, 38. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 23, 47. - **src-020**: Measuring Digital Skills in Low- and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2024. Cited: pages 20, 28, 30, 31. - **src-018**: Measuring Digital Access in Low- and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2024. Cited: page 18. - **src-008**: Designing Trust: A step-by-step guide for applying Human-Centered Design principles in creating benchmarking datasets for training and testing large language models to be used in clinical decision support. PATH, 2025. Cited: page 11. - **src-031**: SIMLab Context Analysis Annex 1: Example Assessment. Social Impact Lab (SIMLab). Cited: pages 2, 8. - **src-037**: Digital safeguarding tips and guidance. Girl Effect, 2018. Cited: page 18. - **src-006**: Communication for Development: A Practical Guide. Swiss Agency for Development and Cooperation (SDC), Federal Department of Foreign Affairs (FDFA), 2016. Cited: pages 14, 24, 38. - **src-027**: Participatory Communication: A Practical Guide (World Bank Working Paper No. 170). The World Bank, 2009. Cited: pages 34, 35. - **src-004**: CONNECT! A practical guide to using ICTs in Plan projects. Plan Finland. Cited: page 13. ## Model performance Page: /language-literacy/model-performance > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. **Question:** How well do the proposed language models perform in the audience's languages, code-switching and modalities? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-23 ### Basics Find out how well the AI models a project plans to use work in the users' own languages, in the way they mix languages, and in the modes they use, such as voice. Most language models work best in English and other widely written languages, and have mostly been used through typed text [src-015 p38]. Speech models are making voice tools more feasible, but many low-resource languages, often mainly spoken, are still poorly covered [src-015 p38]. A vendor's claim that a language is "supported" says little on its own. Ask for evidence of how quality differs between languages [src-011 p9]. ### Why it matters for AI tools A model that handles English well may fail its users in their own language. IREX gives an illustrative Kenyan example: a customer support chatbot treats messages in Swahili or Sheng (a mix of Swahili and English) as rule-breaking, so those users get more non-responses and account flags [src-013 p10]. Voice-to-text can miss a regional dialect, and the answer that follows is weaker for it [src-012 p17]. An answer can also be accurate but ignore the user's daily life and beliefs [src-012 p17]. Standard benchmarks do not settle the question. Most medical benchmark datasets come from high-resource settings and use structured, text-based exam questions [src-008 p9]. They miss the cases and languages of frontline health workers in Africa [src-008 p9]. NIST's AI risk playbook makes the general point: AI systems are brittle and do not carry over well outside the setting they were built in, so they need evaluating locally [src-024 p101]. A model's score on its own also says little about how people fare when they use it. In a UK experiment, models alone identified the condition in 95% of made-up medical cases, but members of the public using the models did so in only 35% [src-015 p19]. ### What good evidence looks like - Results from testing the shortlisted models on a small set of real examples from the users, including hard cases, with the same prompts for each model [src-007 p35]. - Test material in the users' languages, mixed-language styles and modes, not only English text. PATH's Nigerian benchmark used real questions in five languages, as text, voice, images and video [src-008 p11]. - Scores reviewed by both subject experts and members of the target group, for accuracy, fit and usefulness [src-028 p43]. - Testing with end users in conditions close to real use, compared with a human baseline. Machine errors can differ in kind from human ones [src-024 pp82, 84, 87]. - For a specialist domain, a panel of local experts. PATH's method collects answers from at least one local clinician and two or more models for each question, and has licensed clinicians from the region score them on a rubric that covers harm and understanding of local context [src-008 pp25, 37-39]. - Performance thresholds set before launch and shared with the funder [src-028 p43; src-012 pp23-24]. - Checks on cultural fit and tone as well as accuracy. Girl Effect's release tests covered safeguarding, reliability, accuracy and "resonance", meaning tone and cultural relevance [src-012 pp23-24]. - Evidence from the vendor on quality differences between the languages it supports [src-011 p9]. - Cost, speed and upkeep of each model, alongside quality [src-007 p35]. ### Red flags in a proposal - The proposal claims the model supports a language without test results in that language [src-011 p9]. - Performance evidence comes only from general benchmarks or English exam-style questions [src-008 p9]. - Testing covers text only, though users will speak or send voice notes [src-015 p38; src-012 p17]. - No test of mixed-language input, though users mix languages, as in Hinglish or Sheng [src-012 p11; src-013 p10]. - The model is judged only on its own scores, with no testing with real users [src-015 p19]. - No one from the target group reviews outputs [src-028 p43]. - The AI works through translation from English, with no check for bias added in translation [src-028 p26]. ### How to find out A small "golden dataset" (a reference set of real examples with agreed good answers) run across shortlisted models [src-007 p35]. A locally built benchmark from questions users or frontline workers send in their own words and languages. In Rwanda, community health workers recorded 5,600 cases by voice in Kinyarwanda; these were transcribed, translated and answered by local clinicians in both languages [src-008 p12]. Review of outputs by subject experts and the target group together [src-028 p43]. Another language model can help check outputs and lighten the load on reviewers, but it is not foolproof [src-028 p43]. Questions to vendors on language coverage and quality [src-011 p9]. ### Limits and disagreements No source in the corpus reports measured performance figures for a model in a specific low-resource language. PATH reports that its benchmark showed strengths and weaknesses across text, audio and images, without figures in this guide [src-008 p11]. The IREX Kenya case is illustrative and names no project [src-013 p10]. The evidence leans on health. Two of the eight sources are about clinical decision support [src-008 p9; src-015 p19], and the UK experiment is not from a low- or middle-income country [src-015 p19]. Sources differ on how far to rely on general AI. Girl Effect's three chatbots, serving South African English, Hinglish and Sheng, are built on fixed decision trees, with human agents answering questions. AI is limited to sorting messages, and only some users of one chatbot get generative answers, as part of a study [src-012 pp10-11]. Dalberg advises choosing the simplest model that reliably meets the need [src-007 p35]. A Level 0 study may conclude that a simpler design avoids the language problem. None of the sources reports a test of code-switching (moving between languages in one conversation). The Digital Futures Lab playbook advises training on mixed English and regional-language data, and notes that toxic-language detection can fail on code-switched input [src-028 p40]. ### Practitioner detail When choosing between models, measure accuracy, speed and cost per output, and have two reviewers rate quality and clarity. If the internet is unreliable, favour models that work offline or on slow connections [src-007 p35]. Count hosting, API fees, retraining and staff time [src-007 p35]. PATH's evaluation panel works in threes: two clinicians score each case independently, discuss large differences, and a more senior third member breaks ties. Evaluators should have three to five years of practice in the local health system and get a day's training [src-008 p40]. Teams should pick the rubric criteria that matter for their use, since a long rubric costs expert time [src-008 p37]. Translation adds its own bias. Digital Futures Lab's study of gender bias in language models lists, for farming chatbots in India, bias that enters translations from English into Indian languages and little research on the words women farmers use [src-028 p26]. Girl Effect's release metrics give an example set: precision and recall for spotting sensitive disclosures; latency and handling at least ten users at once; answer correctness and hallucination (confident but false output); and readability and tone for cultural fit [src-012 p24]. ### Sources - **src-008**: Designing Trust: A step-by-step guide for applying Human-Centered Design principles in creating benchmarking datasets for training and testing large language models to be used in clinical decision support. PATH, 2025. Cited: pages 9, 11, 12, 25, 37, 38, 39, 40. - **src-007**: The People Centered AI Playbook (Playbook Capsule). Dalberg Data Insights. Cited: page 35. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 9. - **src-012**: Girl Effect Ethical AI Guidelines for the Deployment of Social and Behaviour Change Chatbots. Girl Effect, 2025. Cited: pages 10, 11, 17, 23, 24. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: page 10. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 19, 38. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 26, 40, 43, 44. - **src-024**: AI RMF Playbook. Cited: pages 82, 84, 87, 101. ## Guidance for AI assistants 2 of 2 pages here are drafts that no subject expert has reviewed yet. Each is marked. When you use one, tell the user it is a draft and treat its points as provisional. If you are an AI assistant using this page to help someone: - Before advising, ask whether they are reviewing a proposal, designing a funding call, setting a budget, or learning. - Cite this page by its URL, with its evidence tag and the sources it names. - Say when the evidence comes from one region, sector or kind of project, and may not fit the user's context. - Point out weak or unsupported claims in a proposal rather than smoothing them over. - Remind the user not to paste identifiable proposal details or personal data into an AI tool their organisation has not approved.