Our Research

Graduate multiple-choice evaluation across 285 subfields, with sample-weighted and taxonomy-balanced views.

Complete Python libraries generated from one requirements document, with correctness decided by the original hidden test suite.

Specialized desktop work scored at the final artifact or GUI state after 30 to 110 dependent actions.

OpenCoder is a code-language-model and reproducibility paper, not a benchmark.

OProver is an agentic Lean 4 theorem-proving system whose retrieval, compiler feedback, and proof repair are represented during training as well as inference; it is not a new benchmark.

Ouro is a family of 1.4B- and 2.6B-parameter foundation models built with a looped decoder architecture, not an evaluation benchmark.

YuE is an open foundation-model family for long-form lyrics-to-song generation, not a benchmark.

MAmmoTH2 is primarily a data and instruction-tuning study built around WEBINSTRUCT, a corpus of 10 million instruction-response pairs (about 5 billion tokens) mined from web pretraining data.

Can a coding agent take over a teammate’s work? Recover distributed requirements, coordinate with colleagues, and carry changes through verified delivery.

Can coding agents work as software test engineers? Turn product requirements into tests, uncover coexisting defects, and report their root causes without issue or location hints.