Agent Evals & SWE-bench Benchmarking

1. Concept Overview & Systemic Problem

The greatest danger in developing agent systems in 2026 is the so-called Behavioral Regression:

• You modify a single sentence in the system prompt to improve Markdown formatting, but unexpectedly the agent stops calling the file reading tool.
• You upgrade from model version v1.2 to v1.3, and the agent starts executing twice as many erroneous shell commands.
• Without automated tests, agent projects devolve into "shamanism," where developers test system quality with two or three manual queries in chat.

Agent Evals is an engineering discipline that transfers classic TDD and CI/CD principles into the realm of nondeterministic language agents. It allows for quantitative measurement of success (Pass Rate), cost in tokens (Cost per Task), and execution time (Latency).

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 AGENT EVALUATION TAXONOMY                   │
├─────────────────────────────────────────────────────────────┤
│ 1. State-Based Evals:                                        │
│    • Unit / Integration Tests Pass (pytest, vitest == 0)    │
│    • File System Diff Verification (were required files created) │
├─────────────────────────────────────────────────────────────┤
│ 2. Trajectory-Based Evals:                                   │
│    • Tool Call Accuracy (was the correct tool called)       │
│    • Step Efficiency (how many steps to achieve the goal)    │
│    • Redundant Action Penalty (penalties for repetitions)    │
├─────────────────────────────────────────────────────────────┤
│ 3. LLM-as-a-Judge Evals:                                    │
│    • Response Tone & Adherence to Style Guides              │
│    • Completeness & Safety Compliance                       │
├─────────────────────────────────────────────────────────────┤
│ 4. Resource & Economic Evals:                               │
│    • Token Burn Rate per Resolved Issue                     │
│    • Wall-Clock Execution Time                              │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

A typical agent evaluation framework (e.g., Inspect AI, Braintrust, or a custom Docker Harness) operates through the following cycle:

1. Test Environment Reset: Launch an isolated Docker container with the initial commit of the codebase where the issue is still present.
2. Task Provisioning: The agent is sent only the bug description (issue.md).
3. Agent Trajectory Recording: The agent executes commands, reads files, and makes modifications. All steps are recorded in OpenTelemetry / JSONL format (action trajectory).
4. Harness Evaluation:
   • The orchestrator retrieves the generated git diff.
   • Hidden verification tests (Fail-to-Pass tests) are executed.
   • The status is recorded as RESOLVED, FAILED, TIMEOUT, or OUT_OF_BUDGET.
5. Metric Aggregation: The overall percentage of resolved tasks (Resolve Rate) for the agent release is computed.

4. Production Engineering Scenarios

01. CI/CD Gate for Updating Agent Prompts

Before merging changes to the .cursorrules file or system prompt in the repository, a GitHub Action runs 30 synthetic tests. If the agent resolves less than 90% of tasks, the pull request is automatically blocked.

02. Model Provider Selection Based on ROI

The team evaluates whether to pay $15 per million tokens for a top model or if a new open model at $0.50 suffices. Running evals on their own 100 tasks reveals precise figures: the top model resolves 78% of tasks, the cheaper one 74%, but costs 20 times less.

03. Continuous Evaluation for Agent Reliability

Implementing a continuous evaluation framework ensures that any changes to the agent's architecture are immediately tested against a comprehensive suite of synthetic tasks, maintaining reliability and performance metrics throughout the development lifecycle.

5. Pitfalls, Common Mistakes & Security

• Contamination of Training Data: Public benchmarks (e.g., baseline HumanEval) may end up in the training data of fresh models, causing the model to "remember" the correct answer but fail to reason. Use only proprietary closed evals or SWE-bench Verified with continuously updated tasks.
• Flaky Evals: The nondeterminism of model temperature can yield 80% success in one run and 65% in another. It is essential to conduct at least 3-5 runs for each task (Pass@k metric).
• Blind Trust in LLM-as-a-Judge: Using another model to evaluate code often suffers from bias towards longer or more aesthetically pleasing responses. Prefer deterministic compilers and tests.