evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksCoding

LiveCodeBench Pro

A live competitive-programming benchmark that rates LLMs with a Codeforces-style Elo on fresh contest problems.

CodingCodeforces EloHigher is better
ScoresAboutRun this benchmark

LiveCodeBench Pro is a competitive-programming eval where your model generates C++ solutions that are judged on real testcases. The official harness (GavinZhengOI/LiveCodeBench-Pro) runs locally: you implement an LLMInterface subclass, run python benchmark.py, and it loads problems from Hugging Face, calls your model, extracts C++ from responses, and grades each submission in a Docker-based local judge (LightCPVerifier + go-judge), producing benchmark_result.json plus accuracy-by-difficulty stats. Important: the local harness yields judge verdicts and pass/accuracy, NOT the headline contest-relative Codeforces Elo, which the maintainers compute from LIVE contests for the public leaderboard. Keep attached to any score: the exact HF dataset snapshot/split, the LightCPVerifier submodule commit, your model name/version, sampling params, and that it is a local-judge pass-rate run (not the live Elo).

Benchmark
LiveCodeBench Pro
Repository
github.com/GavinZhengOI/LiveCodeBench-Pro
Dataset
huggingface.co/datasets/QAQAQAQAQ/LiveCodeBench-Pro
Metric
Codeforces Elo

1Install

shell
git clone --recurse-submodules https://github.com/GavinZhengOI/LiveCodeBench-Pro.git
shell
cd LiveCodeBench-Pro
shell
git submodule update --init --recursive
shell
pip install -r requirements.txt   # or: uv sync
shell
docker --version
shell
sudo usermod -aG docker $USER   # Linux: grant Docker permissions, then re-login
shell
huggingface-cli login   # required: the QAQAQAQAQ/LiveCodeBench-Pro problem dataset is gated; accept its conditions on the HF page first

2Run evaluation

shell
# 1) Implement your model: subclass LLMInterface in api_interface.py and override call_llm(self, user_prompt) -> (response_text, metadata). ExampleLLM (OpenAI gpt-4o) is the reference.
# 2) Edit benchmark.py: set llm_instance = YourLLM() (default is api_interface.ExampleLLM()) and set worker <= physical CPU cores (default 8).
# 3) Run (builds the LightCPVerifier Docker image on first run, starts judge on port 8081, downloads testcases on demand from QAQAQAQAQ/LiveCodeBench-Pro-Testcase):
python benchmark.py

3Expected output

benchmark_result.json in the repo root: a list of per-problem BenchmarkResult records (problem_id, problem_title, difficulty, platform, text_response, code, judge_result, response_meta). benchmark.py also prints BENCHMARK STATISTICS via print_stats(dataset, problem_set): accepted/total and accuracy (%) broken down by split and difficulty, plus per-verdict counts. This is a local-judge pass-rate result; it does NOT compute the contest-relative Codeforces Elo (that is leaderboard-only, derived by maintainers from live contests). Do not report local pass-rate as the Elo metric.

4Submit results

Scoring is folded into the run (benchmark.py prints stats and writes benchmark_result.json). To appear on the public leaderboard, email benchmark_result.json to zz4242@nyu.edu including LLM name and version, any relevant details, and contact info. Always keep attached to a reported number: the model name/version and sampling settings, the HF dataset snapshot for QAQAQAQAQ/LiveCodeBench-Pro (and the testcase repo), the LightCPVerifier submodule commit, worker count, and an explicit note that this is the local-judge pass/accuracy result vs. the live-contest Codeforces Elo shown on livecodebenchpro.com.

Gotchas

You MUST clone with submodules (git clone --recurse-submodules or git submodule update --init --recursive): judge.py builds its Docker image from the LightCPVerifier/ submodule (REPO_DIR = os.path.realpath('LightCPVerifier'), IMAGE_NAME = 'lightcpverifier'). An empty submodule dir makes the image build fail. The submodule URL is https://github.com/YanagiOrigami/LightCPVerifier.git.
The problem dataset QAQAQAQAQ/LiveCodeBench-Pro is GATED on Hugging Face ('You need to agree to share your contact information to access this dataset'): you must log in and accept its conditions (huggingface-cli login / HF_TOKEN) or load_dataset() will fail. The testcase dataset QAQAQAQAQ/LiveCodeBench-Pro-Testcase is public but ~15.3 GB total and is downloaded per-problem (zip per problem_id) at judge time via hf_hub_download.
Requires Linux (README lists Ubuntu 20.04+, Python 3.12+) with cgroup support and Docker; the container runs privileged with shm_size 4g (go-judge sandbox) on port 8081. It will not work in environments without Docker privileged access; macOS/Windows Docker may not provide the needed sandbox kernel features.
The local harness only outputs judge verdicts + accuracy-by-difficulty; it does NOT produce the headline Codeforces Elo. The contest-relative Elo is computed by the maintainers from LIVE contests for the leaderboard, so a local run is not directly comparable to the site's Elo numbers.
ExampleLLM uses OpenAI gpt-4o and requires OPENAI_API_KEY; the default prompt forces C++ output wrapped in ```cpp fences, and util.extract_longest_cpp_code parses ```cpp blocks (last block containing #include), falling back to locating int main( if no fences are found, so a model that emits neither cpp fences nor a recognizable int main( yields no extractable code (counted as a judge failure).