evals.report
BenchmarksLabsCompareRun guidesIn the wild
MetaLlama

Llama 3.1 405B

Meta · Llama. Released Jul 23, 2024.

Llama 3.1 405B is a model from Meta in the Llama family, released Jul 23, 2024. evals.report tracks 16 reported Llama 3.1 405B benchmark scores across SWE-bench Pro, GPQA Diamond, AIME (OTIS Mock), Artificial Analysis Intelligence Index, Epoch Capabilities Index, MMLU-Pro, GDPval, LiveCodeBench, and 8 more — each shown with its benchmark, metric, source status, and date, and never combined into a single ranking.

Open16 results

Benchmark results 16

Compare this model
BenchmarkCategoryScoreMetricStatusDate
SWE-bench ProCoding11.18%% resolvedOfficialJul 23, 2024Details
GPQA DiamondReasoning50.9%accuracyOfficialJul 23, 2024Details
AIME (OTIS Mock)Reasoning9.7%accuracyOfficialJul 23, 2024Details
Artificial Analysis Intelligence IndexReasoning17.4IndexUnverifiedJul 23, 2024Details
Epoch Capabilities IndexReasoning129.1IndexOfficialJul 23, 2024Details
MMLU-ProReasoning73.2%accuracyVerifiedJul 23, 2024Details
GDPvalAgents255EloOfficialJul 23, 2024Details
LiveCodeBenchCoding30.5%Pass@1UnverifiedJul 23, 2024Details
SciCodeCoding29.9%accuracyUnverifiedJul 23, 2024Details
AA-Omniscience: Knowledge and Hallucination BenchmarkReasoning-17AA-Omniscience IndexOfficialJul 23, 2024Details
EQ-Bench Creative Writing v3Chat preference953EloVerifiedJul 23, 2024Details
AILuminate AI Safety BenchmarkOtherGoodSafety gradeVerifiedJul 23, 2024Details
MASK (Model Alignment between Statements and Knowledge)Other61.40Honesty scoreVerifiedJul 23, 2024Details
BigCodeBenchCoding30.4%calibrated Pass@1VerifiedJul 23, 2024Details
SuperGPQAReasoning43.14%accuracyVerifiedJul 23, 2024Details
Gray Swan Arena (Agent Red-Teaming / Indirect Prompt Injection)Agents5.89%Attack Success Rate (ASR)VerifiedJul 23, 2024Details