Comprehension, Not Only Compliance
How denominators and detectors shape cross-lingual safety gaps for Urdu, Roman Urdu and code-switched prompts
The question
Multilingual safety evaluations usually summarise a model with one number: the gap between its safety in English and in another language. For Urdu, which people mostly type as Roman Urdu or mix with English, a measured gap can come from three very different sources:
The model understood the request and produced harmful content.
The model did not understand, and produced loops, off-topic text or an answer in another language.
The evaluator cannot read the response's language or script, and misclassifies it.
Findings
- 92% → 0%Refusal detection collapses outside English
An English-pattern refusal detector caught 92% of English refusals but 0 of 40 Urdu-script refusals, and only 16–19% in Roman Urdu and code-switched text. Scored this way, the same responses show safety gaps of 42.5–70 points instead of 10–40.
- 39.5 → 26.7 ppDenominator choices move the headline gap
Counting only genuine harmful compliance on all scenario pairs, Qwen3-8B's Roman Urdu gap falls from 39.5 to 26.7 points and is no longer significant after Holm correction; counting the same non-scoreable outputs as unsafe raises it to 56.7.
- 71%Models answer Urdu in a different language
Aya Expanse 8B answers 71% of Roman Urdu prompts (85/120) and 65% of code-switched prompts (78/120) in Hindi script, reproducing what the original evaluation found (20 of 30). Script drift changes what a safety judge has to read.
- Multi-model evaluation with independent judges
Across Qwen3-8B, Aya Expanse 8B and Qwen3.5-9B (1,440 generations), two automated judges agree on harmfulness for 96% of responses but on refusal type for only 50%. Both find gaps for Aya Expanse in Urdu script (+11 to +12 points) and Qwen3-8B in Roman Urdu (+10 to +20 points), and none for Qwen3.5-9B. Tables ↗
Method
Every published number of the original evaluation is recomputed from raw outputs (191 of 191 match). Gaps are re-estimated on the full set of scenario pairs, with bounds that count non-scoreable outputs as safe and as unsafe. Refusal detectors are scored against human-reviewed labels in each language form, and the language and script of every response are recorded. The follow-up study is pre-registered, and its primary test was chosen by simulation. Protocol ↗ Statistical validation ↗
Scope
- Findings 1–3 rest on 30 harmful requests in four forms, written by one author, with two models.
- The multi-model results are labelled by automated judges that do not document Urdu support; human validation is under way.
- Open-weight models of 8–9B parameters only; results may not transfer to larger or proprietary models.
- "Safety" follows an operational rubric (harmful compliance, refusal type, comprehension), not a universal definition.