Run LiveBench
The same run guide is also available from the benchmark detail page.
LiveBench is a contamination-free LLM benchmark with objective ground-truth scoring across six categories (reasoning, math, coding, language, data analysis, instruction following). The canonical harness (github.com/LiveBench/LiveBench) is pip-installable (pip install -e .) and runs locally against any OpenAI-compatible or supported API model: run_livebench.py performs generation, ground-truth judgment, and result display in one pass, producing an aggregate score plus per-category/per-task CSVs. When reporting a score, attach the exact harness commit, the --livebench-release-option date used, and the --bench-name subset, since the question set and metric differ by release.
1Install
git clone https://github.com/LiveBench/LiveBench.gitpython -m venv .venvsource .venv/bin/activatecd LiveBenchpip install -e .cd livebench/code_runner && pip install -r requirements_eval.txt && cd ..export OPENAI_API_KEY=<your-key>2Run evaluation
python download_questions.pypython run_livebench.py --model gpt-4o --bench-name live_bench/coding --livebench-release-option 2024-11-25python run_livebench.py --model gpt-4o --bench-name live_bench --mode parallel --parallel-requests 10 --livebench-release-option 2024-11-253Score output
python gen_ground_truth_judgment.py --bench-name live_bench/coding --question-source jsonlpython show_livebench_result.py --bench-name live_bench --model-list gpt-4o4Expected output
run_livebench.py runs the full pipeline (generate answers -> ground-truth judgment -> show results) and prints the LiveBench leaderboard to the terminal. The category breakdown is written to all_groups.csv and the per-task breakdown to all_tasks.csv (under the livebench working directory). The headline metric is the overall LiveBench score (mean of category scores). Only compare scores computed on the same --livebench-release-option and the same --bench-name subset; do not mix scores across release dates or against the hosted leaderboard if a different release was used.
5Submit results
LiveBench reports results on its public leaderboard (livebench.ai); there is no required online submission to evaluate your own model locally (the maintainers also offer to evaluate models via a GitHub issue or email). To self-report, run the harness and cite: the harness commit/version, the exact --model and any --api-base used, the --livebench-release-option date, the --bench-name scope (full live_bench vs a category), and the resulting overall score plus all_groups.csv/all_tasks.csv. Keep the full scaffold (decode settings, parallelism flags) attached to any reported number.