evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksMultimodal

Video-MME

A comprehensive evaluation benchmark for multimodal LLMs in video analysis, using 900 videos (254 hours) and 2,700 human-annotated multiple-choice QA pairs across short, medium, and long durations, scored by answer accuracy with and without subtitles.

MultimodalaccuracyHigher is better
ScoresAboutRun this benchmark

What is Video-MME?

A comprehensive evaluation benchmark for multimodal LLMs in video analysis, using 900 videos (254 hours) and 2,700 human-annotated multiple-choice QA pairs across short, medium, and long durations, scored by answer accuracy with and without subtitles. evals.report tracks reported Video-MME scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported Video-MME score: Kimi K2.5 — 87.4% (accuracy).

ModelLabScore↓Source modelStatusDate
Kimi K2.5OpenMoonshot AI87.4%—UnverifiedJan 27, 2026Details
Gemini 2.5 ProGoogle DeepMind84.8%—VerifiedMar 25, 2025Details
Qwen 3.6 PlusAlibaba / Qwen84.2%—UnverifiedApr 2, 2026Details
Gemini 1.5 ProGoogle DeepMind75.0%—OfficialFeb 15, 2024Details
GPT-4oOpenAI71.9%—OfficialMay 13, 2024Details
Claude 3.5 SonnetAnthropic60.0%—OfficialJun 20, 2024Details

Each row reports the model’s accuracy on Video-MME. Click a row for the full run context.