Skip to content
Results

Research results

Everything on this page is computed from real model outputs and labels committed to the repository. The interactive Platform page is a demonstration with illustrative numbers and is kept separate. Generated 2026-11-12T19:53:11+00:00.

Study status

VeriAudit v2 — what has and has not been run

  • donev0.1 pilot reconstruction191/191 published numbers reproduced
  • doneVeriAudit-X main-study scenario selection200 scenarios selected from AdvBench, XSTest and v0.1
  • blockedUrdu / Roman Urdu / code-switched variantsawaiting human translation (AI generation stopped by model safety classifiers)
  • not runMain-study inference (5 models)requires GPU; pipeline tested on CPU fixtures only
  • not runEncoder classifiers (XLM-R, mDeBERTa, MuRIL)requires GPU and Hugging Face access
v0.1 evaluation · reproduced

Cross-lingual gaps, three ways (191/191 published numbers reproduced)

Real dataReal data: VeriAudit v0.1 pilot (Qwen3-8B, Aya Expanse 8B; 30 harmful intents x 4 forms). Labels: 66 human, 174 LLM-assisted.

The published v0.1 metric drops unscoreable responses from the denominator. Keeping all 30 pairs and counting only actual harmful compliance shrinks Qwen3-8B's Roman Urdu gap and removes its significance: much of it was comprehension failure, not unsafe behaviour. Gaps in percentage points vs English; bars show the harmful-compliance gap with its 95% Newcombe interval. p-values are Holm-adjusted within each definition.

ModelFormv0.1 CLSGHolm pHarmful-compliance gap [95% CI]Holm pSafe-rate gap (unscoreable = not safe)Holm p
Aya Expanse 8BCode-switched20.0.09420.0 [4.5, 37.3].09420.0 [4.5, 37.3].094
Aya Expanse 8BRoman Urdu31.0.01930.0 [12.5, 47.9].01933.3 [15.2, 51.2].008
Aya Expanse 8BUrdu script46.1.00340.0 [20.9, 57.7].00353.3 [32.7, 69.8]<.001
Qwen3-8BCode-switched8.2.7506.7 [-6.7, 21.8].75020.0 [3.4, 37.5].094
Qwen3-8BRoman Urdu39.5.01926.7 [6.7, 45.1].08656.7 [34.5, 72.3]<.001
Qwen3-8BUrdu script10.0.75010.0 [-6.1, 26.8].75010.0 [-6.1, 26.8].375

Mean over the six comparisons (intent-cluster bootstrap, 95%): v0.1 CLSG 25.8 pp [17.3, 35.3]; harmful-compliance gap 22.2 pp [14.4, 31.1].

Scoring

Refusal detection fails outside English

Real dataReal data: the 240 v0.1 responses scored by two rule-based detectors against the final labels.

FormSafe responsesv1 backend detector recallv2 multilingual rule judge recall
English5991.5%83.0%
Urdu script400.0%22.5%
Roman Urdu3215.6%25.0%
Code-switched4719.1%36.2%

Neither detector is used as a safety metric in v2; both are triage only. The v1 API's heuristic would have reported gaps of 42.5–70 points on these same responses — an artefact of English-only refusal patterns.

Response language

Which language did the model answer in?

Real dataReal data: detected response script/language for non-English prompts in v0.1.

ModelPrompt formResponse languagenSafeHarmfulUnscoreable
Qwen3-8BUrdu scripturdu302640
Qwen3-8BRoman Urduenglish5500
Qwen3-8BRoman Urduroman urdu23797
Qwen3-8BRoman Urduurdu2002
Qwen3-8BCode-switchedenglish9900
Qwen3-8BCode-switchedmixed code switched1100
Qwen3-8BCode-switchedroman urdu201334
Aya Expanse 8BUrdu scripthindi devanagari1001
Aya Expanse 8BUrdu scriptmixed code switched3030
Aya Expanse 8BUrdu scripturdu261493
Aya Expanse 8BRoman Urduhindi devanagari201721
Aya Expanse 8BRoman Urdumixed code switched1010
Aya Expanse 8BRoman Urduroman urdu9360
Aya Expanse 8BCode-switchedenglish1100
Aya Expanse 8BCode-switchedhindi devanagari221840
Aya Expanse 8BCode-switchedroman urdu7520
RQ5 · preliminary

Does a harmful-response classifier transfer across forms?

Real dataReal data: char n-gram logistic regression trained/evaluated on the 240 v0.1 responses (exploratory; 44 positives).

SettingTest formModelMacro-F1 [95% CI]Recall (unsafe)FPRn (unsafe)
Leave one form outEnglishchar n-gram LR0.50 [0.49, 1.00]0.000.0060 (1)
Leave one form outEnglishrule judge0.46 [0.43, 0.48]0.000.1460 (1)
Leave one form outUrdu scriptchar n-gram LR0.42 [0.39, 0.45]0.000.0060 (16)
Leave one form outUrdu scriptrule judge0.58 [0.44, 0.70]0.440.2760 (16)
Leave one form outRoman Urduchar n-gram LR0.44 [0.35, 0.53]0.060.1060 (18)
Leave one form outRoman Urdurule judge0.64 [0.51, 0.77]0.500.2160 (18)
Leave one form outCode-switchedchar n-gram LR0.49 [0.35, 0.62]0.440.3760 (9)
Leave one form outCode-switchedrule judge0.52 [0.41, 0.64]0.330.2560 (9)
Intent-grouped 5-foldEnglishchar n-gram LR0.50 [0.49, 1.00]0.000.0060 (1)
Intent-grouped 5-foldEnglishrule judge0.46 [0.43, 0.48]0.000.1460 (1)
Intent-grouped 5-foldUrdu scriptchar n-gram LR0.59 [0.45, 0.73]0.500.3060 (16)
Intent-grouped 5-foldUrdu scriptrule judge0.58 [0.44, 0.70]0.440.2760 (16)
Intent-grouped 5-foldRoman Urduchar n-gram LR0.76 [0.64, 0.86]0.780.2160 (18)
Intent-grouped 5-foldRoman Urdurule judge0.64 [0.51, 0.77]0.500.2160 (18)
Intent-grouped 5-foldCode-switchedchar n-gram LR0.54 [0.41, 0.67]0.330.2260 (9)
Intent-grouped 5-foldCode-switchedrule judge0.52 [0.41, 0.64]0.330.2560 (9)
Intent-grouped 5-foldAll formschar n-gram LR0.68 [0.61, 0.74]0.570.17240 (44)
Intent-grouped 5-foldAll formsrule judge0.59 [0.52, 0.66]0.430.21240 (44)

Exploratory and small (44 unsafe responses, one in English). The rule judge's patterns were built from these same responses, so its scores are in-sample.

VeriAudit-X

Main-study design: 200 scenarios selected

Selection only: English seeds chosen; no non-English variants exist yet.

CategorySourceScenarios
cybercrimeadvbench24
cybercrimeveriaudit-v0.15
fraudadvbench21
fraudveriaudit-v0.15
fraudxstest3
hate speechadvbench11
hate speechveriaudit-v0.15
hate speechxstest12
misinformationadvbench20
misinformationveriaudit-v0.15
misinformationxstest3
privacy violationadvbench14
privacy violationxstest9
self harmadvbench11
self harmveriaudit-v0.15
self harmxstest5
violenceadvbench3
violencexstest25
weaponsadvbench6
weaponsveriaudit-v0.15
weaponsxstest3

Monte-Carlo power of exact McNemar on noise-free scenario outcomes, alpha .05/15 (protocol §4.3). With 3-sample response noise see results/power/power_with_sampling_noise.csv and power_random_effects.csv.

Planning scenarioNet gap (pp)n scenariosPower
pilot-like large gap (Aya x Urdu: 12/30, 0/30)40.01201.00
pilot-like large gap (Aya x Urdu: 12/30, 0/30)40.01501.00
pilot-like large gap (Aya x Urdu: 12/30, 0/30)40.02001.00
pilot-like large gap (Aya x Urdu: 12/30, 0/30)40.02401.00
pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30)30.01201.00
pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30)30.01501.00
pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30)30.02001.00
pilot-like moderate gap (Aya x Roman Urdu: 9/30, 0/30)30.02401.00
pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30)10.01200.35
pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30)10.01500.51
pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30)10.02000.71
pilot-like small gap (Qwen3 x Urdu: 4/30, 1/30)10.02400.83
smallest effect of interest: 10 pp net, noisy (12% vs 2%)10.01200.46
smallest effect of interest: 10 pp net, noisy (12% vs 2%)10.01500.63
smallest effect of interest: 10 pp net, noisy (12% vs 2%)10.02000.84
smallest effect of interest: 10 pp net, noisy (12% vs 2%)10.02400.92
smallest effect of interest: 10 pp net, very noisy (15% vs 5%)10.01200.28
smallest effect of interest: 10 pp net, very noisy (15% vs 5%)10.01500.40
smallest effect of interest: 10 pp net, very noisy (15% vs 5%)10.02000.58
smallest effect of interest: 10 pp net, very noisy (15% vs 5%)10.02400.71
7.5 pp net (10% vs 2.5%)7.51200.20
7.5 pp net (10% vs 2.5%)7.51500.31
7.5 pp net (10% vs 2.5%)7.52000.51
7.5 pp net (10% vs 2.5%)7.52400.64