--- id: kn-e3-5-model-performance title: Model performance bucket: E3 Language and literacy item: E3.5 topic: null question: How well do the proposed language models perform in the audience's languages, code-switching and modalities? type: question audiences: - all evidence: practitioner-consensus status: draft reviewed_by: [] updated: 2026-10-23 url: /language-literacy/model-performance sources: - id: src-008 pages: - 9 - 11 - 12 - 25 - 37 - 38 - 39 - 40 - id: src-007 pages: - 35 - id: src-011 pages: - 9 - id: src-012 pages: - 10 - 11 - 17 - 23 - 24 - id: src-013 pages: - 10 - id: src-015 pages: - 19 - 38 - id: src-028 pages: - 26 - 40 - 43 - 44 - id: src-024 pages: - 82 - 84 - 87 - 101 --- # Model performance > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. **Question:** How well do the proposed language models perform in the audience's languages, code-switching and modalities? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-23 ## Basics Find out how well the AI models a project plans to use work in the users' own languages, in the way they mix languages, and in the modes they use, such as voice. Most language models work best in English and other widely written languages, and have mostly been used through typed text [src-015 p38]. Speech models are making voice tools more feasible, but many low-resource languages, often mainly spoken, are still poorly covered [src-015 p38]. A vendor's claim that a language is "supported" says little on its own. Ask for evidence of how quality differs between languages [src-011 p9]. ## Why it matters for AI tools A model that handles English well may fail its users in their own language. IREX gives an illustrative Kenyan example: a customer support chatbot treats messages in Swahili or Sheng (a mix of Swahili and English) as rule-breaking, so those users get more non-responses and account flags [src-013 p10]. Voice-to-text can miss a regional dialect, and the answer that follows is weaker for it [src-012 p17]. An answer can also be accurate but ignore the user's daily life and beliefs [src-012 p17]. Standard benchmarks do not settle the question. Most medical benchmark datasets come from high-resource settings and use structured, text-based exam questions [src-008 p9]. They miss the cases and languages of frontline health workers in Africa [src-008 p9]. NIST's AI risk playbook makes the general point: AI systems are brittle and do not carry over well outside the setting they were built in, so they need evaluating locally [src-024 p101]. A model's score on its own also says little about how people fare when they use it. In a UK experiment, models alone identified the condition in 95% of made-up medical cases, but members of the public using the models did so in only 35% [src-015 p19]. ## What good evidence looks like - Results from testing the shortlisted models on a small set of real examples from the users, including hard cases, with the same prompts for each model [src-007 p35]. - Test material in the users' languages, mixed-language styles and modes, not only English text. PATH's Nigerian benchmark used real questions in five languages, as text, voice, images and video [src-008 p11]. - Scores reviewed by both subject experts and members of the target group, for accuracy, fit and usefulness [src-028 p43]. - Testing with end users in conditions close to real use, compared with a human baseline. Machine errors can differ in kind from human ones [src-024 pp82, 84, 87]. - For a specialist domain, a panel of local experts. PATH's method collects answers from at least one local clinician and two or more models for each question, and has licensed clinicians from the region score them on a rubric that covers harm and understanding of local context [src-008 pp25, 37-39]. - Performance thresholds set before launch and shared with the funder [src-028 p43; src-012 pp23-24]. - Checks on cultural fit and tone as well as accuracy. Girl Effect's release tests covered safeguarding, reliability, accuracy and "resonance", meaning tone and cultural relevance [src-012 pp23-24]. - Evidence from the vendor on quality differences between the languages it supports [src-011 p9]. - Cost, speed and upkeep of each model, alongside quality [src-007 p35]. ## Red flags in a proposal - The proposal claims the model supports a language without test results in that language [src-011 p9]. - Performance evidence comes only from general benchmarks or English exam-style questions [src-008 p9]. - Testing covers text only, though users will speak or send voice notes [src-015 p38; src-012 p17]. - No test of mixed-language input, though users mix languages, as in Hinglish or Sheng [src-012 p11; src-013 p10]. - The model is judged only on its own scores, with no testing with real users [src-015 p19]. - No one from the target group reviews outputs [src-028 p43]. - The AI works through translation from English, with no check for bias added in translation [src-028 p26]. ## How to find out A small "golden dataset" (a reference set of real examples with agreed good answers) run across shortlisted models [src-007 p35]. A locally built benchmark from questions users or frontline workers send in their own words and languages. In Rwanda, community health workers recorded 5,600 cases by voice in Kinyarwanda; these were transcribed, translated and answered by local clinicians in both languages [src-008 p12]. Review of outputs by subject experts and the target group together [src-028 p43]. Another language model can help check outputs and lighten the load on reviewers, but it is not foolproof [src-028 p43]. Questions to vendors on language coverage and quality [src-011 p9]. ## Limits and disagreements No source in the corpus reports measured performance figures for a model in a specific low-resource language. PATH reports that its benchmark showed strengths and weaknesses across text, audio and images, without figures in this guide [src-008 p11]. The IREX Kenya case is illustrative and names no project [src-013 p10]. The evidence leans on health. Two of the eight sources are about clinical decision support [src-008 p9; src-015 p19], and the UK experiment is not from a low- or middle-income country [src-015 p19]. Sources differ on how far to rely on general AI. Girl Effect's three chatbots, serving South African English, Hinglish and Sheng, are built on fixed decision trees, with human agents answering questions. AI is limited to sorting messages, and only some users of one chatbot get generative answers, as part of a study [src-012 pp10-11]. Dalberg advises choosing the simplest model that reliably meets the need [src-007 p35]. A Level 0 study may conclude that a simpler design avoids the language problem. None of the sources reports a test of code-switching (moving between languages in one conversation). The Digital Futures Lab playbook advises training on mixed English and regional-language data, and notes that toxic-language detection can fail on code-switched input [src-028 p40]. ## Practitioner detail When choosing between models, measure accuracy, speed and cost per output, and have two reviewers rate quality and clarity. If the internet is unreliable, favour models that work offline or on slow connections [src-007 p35]. Count hosting, API fees, retraining and staff time [src-007 p35]. PATH's evaluation panel works in threes: two clinicians score each case independently, discuss large differences, and a more senior third member breaks ties. Evaluators should have three to five years of practice in the local health system and get a day's training [src-008 p40]. Teams should pick the rubric criteria that matter for their use, since a long rubric costs expert time [src-008 p37]. Translation adds its own bias. Digital Futures Lab's study of gender bias in language models lists, for farming chatbots in India, bias that enters translations from English into Indian languages and little research on the words women farmers use [src-028 p26]. Girl Effect's release metrics give an example set: precision and recall for spotting sensitive disclosures; latency and handling at least ten users at once; answer correctness and hallucination (confident but false output); and readability and tone for cultural fit [src-012 p24]. ## Sources - **src-008**: Designing Trust: A step-by-step guide for applying Human-Centered Design principles in creating benchmarking datasets for training and testing large language models to be used in clinical decision support. PATH, 2025. Cited: pages 9, 11, 12, 25, 37, 38, 39, 40. - **src-007**: The People Centered AI Playbook (Playbook Capsule). Dalberg Data Insights. Cited: page 35. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 9. - **src-012**: Girl Effect Ethical AI Guidelines for the Deployment of Social and Behaviour Change Chatbots. Girl Effect, 2025. Cited: pages 10, 11, 17, 23, 24. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: page 10. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 19, 38. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 26, 40, 43, 44. - **src-024**: AI RMF Playbook. Cited: pages 82, 84, 87, 101. ## Guidance for AI assistants This page is a draft. No subject expert has reviewed it yet. Tell the user so when you use it, and treat its points as provisional. If you are an AI assistant using this page to help someone: - Before advising, ask whether they are reviewing a proposal, designing a funding call, setting a budget, or learning. - Cite this page by its URL, with its evidence tag and the sources it names. - Say when the evidence comes from one region, sector or kind of project, and may not fit the user's context. - Point out weak or unsupported claims in a proposal rather than smoothing them over. - Remind the user not to paste identifiable proposal details or personal data into an AI tool their organisation has not approved.