Professional roles we evaluate.
Each role groups papers and benchmarks around concrete tasks in programming, mathematics, machine learning, and audio/video production.
Programmer
Repository-level implementation, code editing, review, testing, and debugging.
A programmer lives inside the repository and the toolchain: requirements arrive through issues and tickets, context flows through Slack threads and design docs, code moves through GitHub — branches, pull requests, reviews — and nothing ships until CI is green. The job demands reading a requirement and turning it into working code, navigating and editing a living codebase without breaking it, writing the tests that prove the change, and debugging until everything passes.
Requirement understanding · Coding & code review · Testing & debugging
Benchmark paper · Repository Generation
NL2Repo-Bench
Complete Python libraries generated from one requirements document, with correctness decided by the original hidden test suite.
Benchmark paper · Knowledge
CodeSimpleQA
Bilingual factual programming questions, scored separately from executable code generation.
Benchmark paper · Code Editing
CodeEditorBench
Execution-checked edits across debugging, translation, optimization, and changing requirements.
Model paper · Open Code LLM
OpenCoder
OpenCoder is a code-language-model and reproducibility paper, not a benchmark.
System paper · Code Execution
OpenCodeInterpreter
OpenCodeInterpreter is a family of open code-generation systems trained to revise programs after compiler output and user feedback, rather than a new benchmark.
Mathematician
Mathematical reasoning, rigorous proof construction, and verification of complex solutions.
A mathematician turns an open-ended question into a precise statement, chooses the right definitions and tools, and builds an argument that survives careful scrutiny. The work moves between exploring examples, identifying patterns, testing conjectures, and checking every assumption in a proof. In AI evaluation, this perspective helps distinguish a plausible-looking answer from a valid solution, trace errors in multi-step reasoning, and connect informal explanations with formally verifiable results.
Problem formulation · Proof & formal verification · Reasoning evaluation
System paper · Formal Theorem Proving
OProver
OProver is an agentic Lean 4 theorem-proving system whose retrieval, compiler feedback, and proof repair are represented during training as well as inference; it is not a new benchmark.
Research paper · Instruction Data
MAmmoTH2
MAmmoTH2 is primarily a data and instruction-tuning study built around WEBINSTRUCT, a corpus of 10 million instruction-response pairs (about 5 billion tokens) mined from web pretraining data.
Benchmark paper · Graduate Knowledge
SuperGPQA
Graduate multiple-choice evaluation across 285 subfields, with sample-weighted and taxonomy-balanced views.
Machine Learning Engineer
Data preparation, model experimentation, and reproducible evaluation across the machine-learning lifecycle.
A machine learning engineer connects a research idea to a dependable experimental pipeline. The work involves examining data quality, choosing baselines, configuring training runs, comparing checkpoints, and investigating why a promising result does or does not generalize. This role brings a practical view of model behavior: experiments should be reproducible, metrics should reflect the task, and improvements should hold up under changing data, compute budgets, and real workflow constraints.
Data & training pipelines · Experiment design · Model & workflow evaluation
Model paper · Looped Language Models
Ouro
Ouro is a family of 1.4B- and 2.6B-parameter foundation models built with a looped decoder architecture, not an evaluation benchmark.
Model paper · Open Code LLM
OpenCoder
OpenCoder is a code-language-model and reproducibility paper, not a benchmark.
Research paper · Instruction Data
MAmmoTH2
MAmmoTH2 is primarily a data and instruction-tuning study built around WEBINSTRUCT, a corpus of 10 million instruction-response pairs (about 5 billion tokens) mined from web pretraining data.
Benchmark paper · Professional Workflows
Workflow-GYM
Specialized desktop work scored at the final artifact or GUI state after 30 to 110 dependent actions.
Audio & Video Editor
Audio production, timeline-based editing, and quality assessment of creative media workflows.
An audio and video editor turns a creative brief into a coherent finished piece. The work combines selecting source material, shaping a timeline, synchronizing sound and picture, balancing levels, and refining transitions, pacing, and delivery settings. Evaluating AI in this setting means looking beyond a single attractive output: edits must follow the brief, preserve continuity, and remain usable in a real production workflow. Both the final artifact and the sequence of tool-based decisions matter.
Timeline editing · Audio & visual quality · Creative workflow evaluation
