SWE-Lancer

An OpenAI benchmark of over 1,400 real freelance software engineering tasks from Upwork (worth $1M in total payouts), where models either implement IC SWE code patches graded by end-to-end tests or act as SWE managers selecting the best technical proposal, measured by task pass rate and dollars earned.