Run DeepSWE
The same run guide is also available from the benchmark detail page.
DeepSWE is a 113-task long-horizon SWE benchmark (TypeScript, Go, Python, JavaScript, Rust) using the Harbor task format with program-based behavioral verifiers. You clone the public deep-swe task repo and run it with the datacurve-pier runner ('pier'), which drives mini-swe-agent (model-agnostic) or CLI agents (claude-code, codex, gemini-cli, opencode) in isolated Docker or Modal sandboxes; each task's verifier runs after the agent and emits a pass/fail, and the aggregate pass rate is the headline % resolved. Keep attached to any score: the agent (mini-swe-agent for leaderboard parity), the exact --model name, the environment (docker vs modal), task count / sample-seed if a subset, and the pier/deep-swe commit, since the official leaderboard uses mini-swe-agent on Modal.
1Install
git clone https://github.com/datacurve-ai/deep-sweuv tool install datacurve-pier2Run evaluation
export ANTHROPIC_API_KEY=...pier run -p deep-swe/tasks --agent mini-swe-agent --model anthropic/claude-opus-4-7# OpenAI model instead:
export OPENAI_API_KEY=...pier run -p deep-swe/tasks --agent mini-swe-agent --model openai/gpt-5.5# Deterministic random subset (e.g. 10 of 113 tasks):
pier run -p deep-swe/tasks --agent mini-swe-agent --n-tasks 10 --sample-seed 0# Single task:
pier run -p deep-swe/tasks/<task-id> --agent mini-swe-agent# Run in parallel Modal sandboxes (leaderboard parity):
pier run -p deep-swe/tasks --agent mini-swe-agent --env modal --env-file .env3Expected output
Per-task trials are written under jobs/<timestamp_or_name>/<trial_id>/. Grading is folded into 'pier run': after the agent finishes, the task's program-based verifier (tests/test.sh with tests/test.patch applied at grading time) exercises the described behavior through public APIs/observable outputs and records pass/fail for that trial. The headline metric is the aggregate pass rate (% resolved) over the 113 tasks, reported with confidence intervals on the official leaderboard (e.g. 'gpt-5.5 [xhigh] 70%±4%', 'claude-opus-4.7 [max] 54%±5%'). The verifier accepts any solution with correct observable behavior; the held-out solution/ reference patch is never used at grading time. Report your own pass rate as benchmark-local and do not compare across different agents/environments or to other SWE benchmarks (e.g. SWE-bench).
4Submit results
There is no public auto-submission flow. To appear on the official DeepSWE leaderboard, contact serena@datacurve.ai with your results (per the /run page). Keep the run context attached to any reported number: agent (use mini-swe-agent for leaderboard comparability), exact --model name and reasoning effort, --env (docker vs modal), whether you ran the full 113-task corpus or a --n-tasks/--sample-seed subset, and the pier + deep-swe commit hashes. Official leaderboard scores were produced with Pier running mini-swe-agent on Modal.