Video-MME
A comprehensive evaluation benchmark for multimodal LLMs in video analysis, using 900 videos (254 hours) and 2,700 human-annotated multiple-choice QA pairs across short, medium, and long durations, scored by answer accuracy with and without subtitles.
What this benchmark measures
A comprehensive evaluation benchmark for multimodal LLMs in video analysis, using 900 videos (254 hours) and 2,700 human-annotated multiple-choice QA pairs across short, medium, and long durations, scored by answer accuracy with and without subtitles.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is accuracy. It should be interpreted within Video-MME, not compared as part of a site-wide ranking.
Frequently asked
What is Video-MME?
A comprehensive evaluation benchmark for multimodal LLMs in video analysis, using 900 videos (254 hours) and 2,700 human-annotated multiple-choice QA pairs across short, medium, and long durations, scored by answer accuracy with and without subtitles. It is a multimodal benchmark measured by accuracy.
What does accuracy mean on Video-MME?
Video-MME reports accuracy (%); higher is better. Scores are shown only within Video-MME and are never averaged with other benchmarks.
What is the top reported Video-MME score?
Kimi K2.5 has the top reported score on Video-MME: 87.4% (accuracy).
Why do Video-MME scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. Video-MME scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".