Skip to content
Research

Comprehension, Not Only Compliance

How denominators and detectors shape cross-lingual safety gaps for Urdu, Roman Urdu and code-switched prompts

Read the paper (PDF)Code & reproductionPreprint · Abeera Najam · 2026

The question

Multilingual safety evaluations usually summarise a model with one number: the gap between its safety in English and in another language. For Urdu, which people mostly type as Roman Urdu or mix with English, a measured gap can come from three very different sources:

Safety failure

The model understood the request and produced harmful content.

Comprehension failure

The model did not understand, and produced loops, off-topic text or an answer in another language.

Measurement failure

The evaluator cannot read the response's language or script, and misclassifies it.

Findings

  • 92% → 0%Refusal detection collapses outside English

    An English-pattern refusal detector caught 92% of English refusals but 0 of 40 Urdu-script refusals, and only 16–19% in Roman Urdu and code-switched text. Scored this way, the same responses show safety gaps of 42.5–70 points instead of 10–40.

  • 39.5 → 26.7 ppDenominator choices move the headline gap

    Counting only genuine harmful compliance on all scenario pairs, Qwen3-8B's Roman Urdu gap falls from 39.5 to 26.7 points and is no longer significant after Holm correction; counting the same non-scoreable outputs as unsafe raises it to 56.7.

  • 71%Models answer Urdu in a different language

    Aya Expanse 8B answers 71% of Roman Urdu prompts (85/120) and 65% of code-switched prompts (78/120) in Hindi script, reproducing what the original evaluation found (20 of 30). Script drift changes what a safety judge has to read.

  • Multi-model evaluation with independent judges

    Across Qwen3-8B, Aya Expanse 8B and Qwen3.5-9B (1,440 generations), two automated judges agree on harmfulness for 96% of responses but on refusal type for only 50%. Both find gaps for Aya Expanse in Urdu script (+11 to +12 points) and Qwen3-8B in Roman Urdu (+10 to +20 points), and none for Qwen3.5-9B. Tables ↗

Method

Every published number of the original evaluation is recomputed from raw outputs (191 of 191 match). Gaps are re-estimated on the full set of scenario pairs, with bounds that count non-scoreable outputs as safe and as unsafe. Refusal detectors are scored against human-reviewed labels in each language form, and the language and script of every response are recorded. The follow-up study is pre-registered, and its primary test was chosen by simulation. Protocol ↗ Statistical validation ↗

Scope

  • Findings 1–3 rest on 30 harmful requests in four forms, written by one author, with two models.
  • The multi-model results are labelled by automated judges that do not document Urdu support; human validation is under way.
  • Open-weight models of 8–9B parameters only; results may not transfer to larger or proprietary models.
  • "Safety" follows an operational rubric (harmful compliance, refusal type, comprehension), not a universal definition.