evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksOther

AILuminate AI Safety Benchmark

MLCommons' standardized AI safety benchmark that grades how often general-purpose chat models produce policy-violating responses across 12 hazard categories (e.g. violent crimes, CSAM, hate, self-harm, specialized advice), assigning an ordinal safety grade from Poor to Excellent relative to a sub-15B open-weight reference system.

OtherSafety gradeHigher is better
ScoresAboutRun this benchmark

What is AILuminate AI Safety Benchmark?

MLCommons' standardized AI safety benchmark that grades how often general-purpose chat models produce policy-violating responses across 12 hazard categories (e.g. violent crimes, CSAM, hate, self-harm, specialized advice), assigning an ordinal safety grade from Poor to Excellent relative to a sub-15B open-weight reference system. evals.report tracks reported AILuminate AI Safety Benchmark scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported AILuminate AI Safety Benchmark score: Claude 3.5 Sonnet — Very Good (Safety grade).

ModelLabScore↓Source modelStatusDate
Claude 3.5 SonnetAnthropicVery Good—VerifiedJun 20, 2024Details
GPT-4oOpenAIGood—VerifiedMay 13, 2024Details
Gemini 1.5 ProGoogle DeepMindGood—VerifiedFeb 15, 2024Details
Gemini 2.0 FlashGoogle DeepMindGood—VerifiedDec 11, 2024Details
Llama 3.1 405BOpenMetaGood—VerifiedJul 23, 2024Details
Mistral LargeMistral AIGood—VerifiedFeb 26, 2024Details

Each row reports the model’s Safety grade on AILuminate AI Safety Benchmark. Click a row for the full run context.