SWE-Lancer
An OpenAI benchmark of over 1,400 real freelance software engineering tasks from Upwork (worth $1M in total payouts), where models either implement IC SWE code patches graded by end-to-end tests or act as SWE managers selecting the best technical proposal, measured by task pass rate and dollars earned.
What this benchmark measures
An OpenAI benchmark of over 1,400 real freelance software engineering tasks from Upwork (worth $1M in total payouts), where models either implement IC SWE code patches graded by end-to-end tests or act as SWE managers selecting the best technical proposal, measured by task pass rate and dollars earned.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is IC SWE pass rate (Diamond). It should be interpreted within SWE-Lancer, not compared as part of a site-wide ranking.