Overview
TestEngineerBench evaluates coding agents as autonomous software test engineers. Given a product requirements document, an unrepaired repository, and a configured environment, an agent decides what to test and where to investigate. No issue reports, target files, functions, or designated entry points are supplied.
Two independent tasks separate test generation from defect discovery and reporting. The key finding is that valid tests and broad code coverage do not guarantee useful defect detection: agents can exercise the right code while failing to check the behavior that distinguishes a defect from its repair.
Real repositories, coexisting defects
The dataset contains 100 instances from 78 open-source Python repositories across 16 application categories. Its 531 recorded defect occurrences are counted separately per instance, not claimed as 531 unique defects. Each instance contains 4–15 coexisting defects, with a median of five.
A typical instance has reference defect locations across five files and 11.5 edit hunks. The product requirements document describes expected functionality, boundary conditions, and error behavior; reference fixes and localization targets remain evaluator-only.
Two tasks, complementary measures
- Test generation: submit a standalone suite, evaluated on both defective and repaired snapshots. Head Pass Rate (HP) measures tests passing after repair; Fail-to-Pass (F2P) measures tests that fail before repair and pass afterward. Entry Coverage and Change-focused Coverage measure execution reach, not defect detection.
- Defect discovery and reporting: independently author and run probes, then submit source-grounded locations and execution evidence. Recall and precision are scored at file, function, and edit-hunk levels, with a three-line tolerance for hunk matching.
- Distinct bug localization requires matching all reference files and function symbols and more than half the reference edit hunks. HP pools test counts; F2P and most other main-table metrics average across instances. Distinct bug rates weight each evaluable bug occurrence equally.
Coverage is not defect detection
Across seven evaluated agents, the manuscript reports mean HP of 91.47% and Change-focused Coverage of 80.48%, but mean F2P of only 26.36%. In a separate inspection of 49,317 registered tests from 500 evaluations across five systems, 67.44% pass on both snapshots without exposing defects. That inspection is a subset, not the full seven-agent evaluation.
| Measure | Reported result | Scope |
|---|---|---|
| Head Pass Rate | 91.47% | Mean across 7 agents |
| Change-focused Coverage | 80.48% | Mean across 7 agents |
| Fail-to-Pass Rate | 26.36% | Mean across 7 agents |
| File / function / hunk recall | 78.11% / 51.29% / 51.88% | Mean across 7 agents |
| Distinct bugs localized | 33.14%–49.04% | Range across agents |
| Multi-file bugs localized | 2.00%–7.84% | Range across agents |
The largest localization gap appears between finding an affected file and isolating the responsible function. Multi-file defects are particularly difficult: recognizing a visible exception is not enough to trace a broken contract across callers and dependent components.
Active testing helps, with trade-offs
A paired GPT-5.6-Sol experiment across 100 instances compares running existing repository tests with authoring additional tests from the requirements. Active test authoring improves file, function, and hunk recall by 4.99, 2.77, and 2.69 percentage points respectively.
The gains are uneven: median paired recall change is zero at all three levels. Mean precision also falls as reported locations grow faster than confirmed defect sites. This is evidence for targeted exploration beyond existing suites, not a guarantee that more generated tests improve every agent or instance.
Toward reliable autonomous testing
Effective test engineering needs requirements-derived assertions, defect-triggering inputs, cross-module investigation, and disciplined evidence selection. Passing tests, larger suites, and higher coverage should be assessed alongside defect discrimination and precise localization, rather than treated as substitutes.
The current benchmark is Python-only and scores reports against reference defect mappings. Localization agreement does not establish independent replay of every submitted report’s evidence; the manuscript names additional languages and independent evidence verification as future extensions. This page summarizes the supplied manuscript dated October 25, 2026, without independent result reproduction or an inferred public paper/code link.