evals.report
BenchmarksLabsCompareRun guides
BenchmarksOther

AILuminate AI Safety Benchmark

MLCommons' standardized AI safety benchmark that grades how often general-purpose chat models produce policy-violating responses across 12 hazard categories (e.g. violent crimes, CSAM, hate, self-harm, specialized advice), assigning an ordinal safety grade from Poor to Excellent relative to a sub-15B open-weight reference system.

OtherSafety gradeHigher is better
ScoresAboutRun this benchmark
ModelLabScore↓Source modelStatusDate
Claude 3.5 SonnetAnthropicVery Good—VerifiedJun 20, 2024Details
GPT-4oOpenAIGood—VerifiedMay 13, 2024Details
Gemini 1.5 ProGoogle DeepMindGood—VerifiedFeb 15, 2024Details
Gemini 2.0 FlashGoogle DeepMindGood—VerifiedDec 11, 2024Details
Llama 3.1 405BMetaGood—VerifiedJul 23, 2024Details
Mistral LargeMistral AIGood—VerifiedFeb 26, 2024Details

Each row reports the model’s Safety grade on AILuminate AI Safety Benchmark. Click a row for the full run context.