Skip to content
Independent research · Open-source evaluation framework

VERIAUDIT

Measuring what multilingual AI safety evaluations actually measure.

When a language model looks less safe in Urdu, is it producing harmful content, failing to understand the request, or being misjudged by an evaluator built for English? VeriAudit separates the three, across English, Urdu script, Roman Urdu and Urdu–English code-switching.

Findings

Cross-lingual safety gaps depend on how they are measured

Three connected measurement problems, each reproducible from the released outputs and code.

92% → 0%

Refusal detection collapses outside English

An English-pattern refusal detector caught 92% of English refusals but 0 of 40 Urdu-script refusals, and only 16–19% in Roman Urdu and code-switched text. Scored this way, the same responses show safety gaps of 42.5–70 points instead of 10–40.

source: results/v1_reconstruction ↗
39.5 → 26.7 pp

Denominator choices move the headline gap

Counting only genuine harmful compliance on all scenario pairs, Qwen3-8B's Roman Urdu gap falls from 39.5 to 26.7 points and is no longer significant after Holm correction; counting the same non-scoreable outputs as unsafe raises it to 56.7.

source: results/v1_reconstruction ↗
71%

Models answer Urdu in a different language

Aya Expanse 8B answers 71% of Roman Urdu prompts (85/120) and 65% of code-switched prompts (78/120) in Hindi script, reproducing what the original evaluation found (20 of 30). Script drift changes what a safety judge has to read.

source: results/v2/exploratory/v01-panel ↗
Multi-model evaluation

Newer models, independent automated judges

Three open-weight models, 1,440 generations (greedy plus three samples per prompt), each response labelled by two automated judges from model families outside the panel. Gap in harmful compliance versus English, percentage points with 95% intervals, per judge.

ModelPrompt formJudge A: Mistral-NemoJudge B: Granite-3.3Note
Aya Expanse 8BUrdu script+12.2 [3.9, 20.5]+11.1 [2.2, 20.1]Holm p = .049 (Nemo)
Qwen3-8BRoman Urdu+10.1 [2.1, 18.2]+20.4 [3.2, 37.6]p < .05 unadjusted, both judges
Qwen3.5-9Ball forms+1.1 to +2.20.0 to +2.8no evidence of a gap
Evaluator reliability

The two judges agree on whether a response is harmful 96% of the time (κ = 0.44), but on how a model refused only 50% of the time. Under a strict two-judge resolution rule most labels stay unresolved, so evaluator agreement is itself a research question, and human validation of the labels is the next step.

Reading these numbers

Both judges see no harmful compliance in English for any model, and the same direction of gap for Aya in Urdu script and Qwen3-8B in Roman Urdu. These are exploratory, automatically judged results on 30 scenarios; one comparison survives multiple-testing correction. Gemma 4 could not be run on the available GPUs.

full tables and run manifest ↗
Open-source framework

Research infrastructure built to be checked

Everything needed to reproduce, audit and extend the results is public.

Exact reproduction

All 191 published numbers of the original evaluation recomputed from raw outputs, as an automated check.

Pre-registered protocol

Hypotheses, estimands, tests and decision rules fixed before data collection, with a dated amendment log.

Validated statistics

Primary test chosen by simulation: 11 null scenarios × 20,000 runs; the common signed-rank test was shown not to test the reported gap.

Integrity safeguards

Hash-bound runs, pinned model revisions, refusal of tampered or incomplete runs; 13 failure scenarios replayed against the code.

Human-validation workflow

Blind, double annotation with third-person adjudication, independence enforced in code, Urdu-safe spreadsheets.

Open evaluation pipeline

Model runners, two-judge labelling, analysis and reporting for English, Urdu script, Roman Urdu and code-switching.

Roadmap

From measurement study to benchmark

  1. Now

    Human validation of the judge labels

    Two independent Urdu readers label a stratified sample; a third adjudicates.

  2. Next

    VeriAudit-X benchmark

    200 scenarios with matched benign contrasts, human-written and independently verified in four forms, plus controlled Roman Urdu and code-switching variants.

  3. Then

    Pre-registered main study

    Calibration run, then the confirmatory comparison of four models with human-validated labels.

Researcher

VeriAudit is designed, built and run by Abeera Najam as independent research on trustworthy multilingual AI: how evaluation methods, language understanding and safety behaviour interact for languages that English-centric tools handle poorly.