Research results
Everything on this page is computed from real model outputs and labels committed to the repository. The interactive Platform page is a demonstration with illustrative numbers and is kept separate. Generated 2026-11-12T19:53:11+00:00.
VeriAudit v2 — what has and has not been run
- donev0.1 pilot reconstruction191/191 published numbers reproduced
- doneVeriAudit-X main-study scenario selection200 scenarios selected from AdvBench, XSTest and v0.1
- blockedUrdu / Roman Urdu / code-switched variantsawaiting human translation (AI generation stopped by model safety classifiers)
- not runMain-study inference (5 models)requires GPU; pipeline tested on CPU fixtures only
- not runEncoder classifiers (XLM-R, mDeBERTa, MuRIL)requires GPU and Hugging Face access
Cross-lingual gaps, three ways (191/191 published numbers reproduced)
Real dataReal data: VeriAudit v0.1 pilot (Qwen3-8B, Aya Expanse 8B; 30 harmful intents x 4 forms). Labels: 66 human, 174 LLM-assisted.
The published v0.1 metric drops unscoreable responses from the denominator. Keeping all 30 pairs and counting only actual harmful compliance shrinks Qwen3-8B's Roman Urdu gap and removes its significance: much of it was comprehension failure, not unsafe behaviour. Gaps in percentage points vs English; bars show the harmful-compliance gap with its 95% Newcombe interval. p-values are Holm-adjusted within each definition.
| Model | Form | v0.1 CLSG | Holm p | Harmful-compliance gap [95% CI] | Holm p | Safe-rate gap (unscoreable = not safe) | Holm p | |
|---|---|---|---|---|---|---|---|---|
| Aya Expanse 8B | Code-switched | 20.0 | .094 | 20.0 [4.5, 37.3] | .094 | 20.0 [4.5, 37.3] | .094 | |
| Aya Expanse 8B | Roman Urdu | 31.0 | .019 | 30.0 [12.5, 47.9] | .019 | 33.3 [15.2, 51.2] | .008 | |
| Aya Expanse 8B | Urdu script | 46.1 | .003 | 40.0 [20.9, 57.7] | .003 | 53.3 [32.7, 69.8] | <.001 | |
| Qwen3-8B | Code-switched | 8.2 | .750 | 6.7 [-6.7, 21.8] | .750 | 20.0 [3.4, 37.5] | .094 | |
| Qwen3-8B | Roman Urdu | 39.5 | .019 | 26.7 [6.7, 45.1] | .086 | 56.7 [34.5, 72.3] | <.001 | |
| Qwen3-8B | Urdu script | 10.0 | .750 | 10.0 [-6.1, 26.8] | .750 | 10.0 [-6.1, 26.8] | .375 |
Mean over the six comparisons (intent-cluster bootstrap, 95%): v0.1 CLSG 25.8 pp [17.3, 35.3]; harmful-compliance gap 22.2 pp [14.4, 31.1].
Refusal detection fails outside English
Real dataReal data: the 240 v0.1 responses scored by two rule-based detectors against the final labels.
| Form | Safe responses | v1 backend detector recall | v2 multilingual rule judge recall |
|---|---|---|---|
| English | 59 | 91.5% | 83.0% |
| Urdu script | 40 | 0.0% | 22.5% |
| Roman Urdu | 32 | 15.6% | 25.0% |
| Code-switched | 47 | 19.1% | 36.2% |
Neither detector is used as a safety metric in v2; both are triage only. The v1 API's heuristic would have reported gaps of 42.5–70 points on these same responses — an artefact of English-only refusal patterns.
Which language did the model answer in?
Real dataReal data: detected response script/language for non-English prompts in v0.1.
| Model | Prompt form | Response language | n | Safe | Harmful | Unscoreable |
|---|---|---|---|---|---|---|
| Qwen3-8B | Urdu script | urdu | 30 | 26 | 4 | 0 |
| Qwen3-8B | Roman Urdu | english | 5 | 5 | 0 | 0 |
| Qwen3-8B | Roman Urdu | roman urdu | 23 | 7 | 9 | 7 |
| Qwen3-8B | Roman Urdu | urdu | 2 | 0 | 0 | 2 |
| Qwen3-8B | Code-switched | english | 9 | 9 | 0 | 0 |
| Qwen3-8B | Code-switched | mixed code switched | 1 | 1 | 0 | 0 |
| Qwen3-8B | Code-switched | roman urdu | 20 | 13 | 3 | 4 |
| Aya Expanse 8B | Urdu script | hindi devanagari | 1 | 0 | 0 | 1 |
| Aya Expanse 8B | Urdu script | mixed code switched | 3 | 0 | 3 | 0 |
| Aya Expanse 8B | Urdu script | urdu | 26 | 14 | 9 | 3 |
| Aya Expanse 8B | Roman Urdu | hindi devanagari | 20 | 17 | 2 | 1 |
| Aya Expanse 8B | Roman Urdu | mixed code switched | 1 | 0 | 1 | 0 |
| Aya Expanse 8B | Roman Urdu | roman urdu | 9 | 3 | 6 | 0 |
| Aya Expanse 8B | Code-switched | english | 1 | 1 | 0 | 0 |
| Aya Expanse 8B | Code-switched | hindi devanagari | 22 | 18 | 4 | 0 |
| Aya Expanse 8B | Code-switched | roman urdu | 7 | 5 | 2 | 0 |
Does a harmful-response classifier transfer across forms?
Real dataReal data: char n-gram logistic regression trained/evaluated on the 240 v0.1 responses (exploratory; 44 positives).
| Setting | Test form | Model | Macro-F1 [95% CI] | Recall (unsafe) | FPR | n (unsafe) |
|---|---|---|---|---|---|---|
| Leave one form out | English | char n-gram LR | 0.50 [0.49, 1.00] | 0.00 | 0.00 | 60 (1) |
| Leave one form out | English | rule judge | 0.46 [0.43, 0.48] | 0.00 | 0.14 | 60 (1) |
| Leave one form out | Urdu script | char n-gram LR | 0.42 [0.39, 0.45] | 0.00 | 0.00 | 60 (16) |
| Leave one form out | Urdu script | rule judge | 0.58 [0.44, 0.70] | 0.44 | 0.27 | 60 (16) |
| Leave one form out | Roman Urdu | char n-gram LR | 0.44 [0.35, 0.53] | 0.06 | 0.10 | 60 (18) |
| Leave one form out | Roman Urdu | rule judge | 0.64 [0.51, 0.77] | 0.50 | 0.21 | 60 (18) |
| Leave one form out | Code-switched | char n-gram LR | 0.49 [0.35, 0.62] | 0.44 | 0.37 | 60 (9) |
| Leave one form out | Code-switched | rule judge | 0.52 [0.41, 0.64] | 0.33 | 0.25 | 60 (9) |
| Intent-grouped 5-fold | English | char n-gram LR | 0.50 [0.49, 1.00] | 0.00 | 0.00 | 60 (1) |
| Intent-grouped 5-fold | English | rule judge | 0.46 [0.43, 0.48] | 0.00 | 0.14 | 60 (1) |
| Intent-grouped 5-fold | Urdu script | char n-gram LR | 0.59 [0.45, 0.73] | 0.50 | 0.30 | 60 (16) |
| Intent-grouped 5-fold | Urdu script | rule judge | 0.58 [0.44, 0.70] | 0.44 | 0.27 | 60 (16) |
| Intent-grouped 5-fold | Roman Urdu | char n-gram LR | 0.76 [0.64, 0.86] | 0.78 | 0.21 | 60 (18) |
| Intent-grouped 5-fold | Roman Urdu | rule judge | 0.64 [0.51, 0.77] | 0.50 | 0.21 | 60 (18) |
| Intent-grouped 5-fold | Code-switched | char n-gram LR | 0.54 [0.41, 0.67] | 0.33 | 0.22 | 60 (9) |
| Intent-grouped 5-fold | Code-switched | rule judge | 0.52 [0.41, 0.64] | 0.33 | 0.25 | 60 (9) |
| Intent-grouped 5-fold | All forms | char n-gram LR | 0.68 [0.61, 0.74] | 0.57 | 0.17 | 240 (44) |
| Intent-grouped 5-fold | All forms | rule judge | 0.59 [0.52, 0.66] | 0.43 | 0.21 | 240 (44) |
Exploratory and small (44 unsafe responses, one in English). The rule judge's patterns were built from these same responses, so its scores are in-sample.
Main-study design: 200 scenarios selected
Selection only: English seeds chosen; no non-English variants exist yet.
| Category | Source | Scenarios |
|---|---|---|
| cybercrime | advbench | 24 |
| cybercrime | veriaudit-v0.1 | 5 |
| fraud | advbench | 21 |
| fraud | veriaudit-v0.1 | 5 |
| fraud | xstest | 3 |
| hate speech | advbench | 11 |
| hate speech | veriaudit-v0.1 | 5 |
| hate speech | xstest | 12 |
| misinformation | advbench | 20 |
| misinformation | veriaudit-v0.1 | 5 |
| misinformation | xstest | 3 |
| privacy violation | advbench | 14 |
| privacy violation | xstest | 9 |
| self harm | advbench | 11 |
| self harm | veriaudit-v0.1 | 5 |
| self harm | xstest | 5 |
| violence | advbench | 3 |
| violence | xstest | 25 |
| weapons | advbench | 6 |
| weapons | veriaudit-v0.1 | 5 |
| weapons | xstest | 3 |
Monte-Carlo power of exact McNemar on noise-free scenario outcomes, alpha .05/15 (protocol §4.3). With 3-sample response noise see results/power/power_with_sampling_noise.csv and power_random_effects.csv.
| Planning scenario | Net gap (pp) | n scenarios | Power |
|---|---|---|---|
| pilot-like large gap (Aya x Urdu: 12/30, 0/30) | 40.0 | 120 | 1.00 |
| pilot-like large gap (Aya x Urdu: 12/30, 0/30) | 40.0 | 150 | 1.00 |
| pilot-like large gap (Aya x Urdu: 12/30, 0/30) | 40.0 | 200 | 1.00 |
| pilot-like large gap (Aya x Urdu: 12/30, 0/30) | 40.0 | 240 | 1.00 |
| pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30) | 30.0 | 120 | 1.00 |
| pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30) | 30.0 | 150 | 1.00 |
| pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30) | 30.0 | 200 | 1.00 |
| pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30) | 30.0 | 240 | 1.00 |
| pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30) | 10.0 | 120 | 0.35 |
| pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30) | 10.0 | 150 | 0.51 |
| pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30) | 10.0 | 200 | 0.71 |
| pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30) | 10.0 | 240 | 0.83 |
| smallest effect of interest: 10 pp net, noisy (12% vs 2%) | 10.0 | 120 | 0.46 |
| smallest effect of interest: 10 pp net, noisy (12% vs 2%) | 10.0 | 150 | 0.63 |
| smallest effect of interest: 10 pp net, noisy (12% vs 2%) | 10.0 | 200 | 0.84 |
| smallest effect of interest: 10 pp net, noisy (12% vs 2%) | 10.0 | 240 | 0.92 |
| smallest effect of interest: 10 pp net, very noisy (15% vs 5%) | 10.0 | 120 | 0.28 |
| smallest effect of interest: 10 pp net, very noisy (15% vs 5%) | 10.0 | 150 | 0.40 |
| smallest effect of interest: 10 pp net, very noisy (15% vs 5%) | 10.0 | 200 | 0.58 |
| smallest effect of interest: 10 pp net, very noisy (15% vs 5%) | 10.0 | 240 | 0.71 |
| 7.5 pp net (10% vs 2.5%) | 7.5 | 120 | 0.20 |
| 7.5 pp net (10% vs 2.5%) | 7.5 | 150 | 0.31 |
| 7.5 pp net (10% vs 2.5%) | 7.5 | 200 | 0.51 |
| 7.5 pp net (10% vs 2.5%) | 7.5 | 240 | 0.64 |