Designing Self-evolving Coding Agents for Terminal-Bench 2.0

Designing Self-evolving Coding Agents for Terminal-Bench 2.0: A Systematic Exploration

Abstract. We present a systematic exploration of designing self-evolving coding agents for Terminal-Bench 2.0, a benchmark of 89 diverse command-line tasks. Starting from a vanilla ReAct agent powered by Claude Opus 4.6 (65.91%), we conduct large-scale controlled experiments across seven groups — establishing baselines, analyzing the performance ceiling (pass@5: 85.88%), exploring four harness components (prompt, memory, tools, skills), refining skill evolution strategies, developing self-verification mechanisms, and decomposing evolution into a multi-agent system (MAS-Evolver). Our investigation reveals that among all mutable components, skill evolution delivers the consistent improvement, and nudged self-verification mechanism further boosts the performance. The MAS-Evolver validates that separating analysis, authoring, and adversarial review into specialized agents produces quality-gated skills (+7.03pp over vanilla). Altogether, our best self-evolution algorithm achieves 76.47%, a +10.56pp improvement over the Opus 4.6 vanilla baseline, ranked top #7 in the terminal bench 2.0 leaderboard.

1. Introduction

1.1 Terminal-Bench 2.0

Terminal-Bench 2.0 (TB2) is a benchmark of 89 diverse command-line tasks spanning system administration, scientific computing, security, data science, and software engineering (E.g., configure web servers, Bayesian networks analysis, ML model training, git history sanitization, and compile software from source). Each task provides a Docker container with a pre-configured environment, a natural language task description, and a hidden test script that validates the solution. Task difficulty ranges widely — from 5-minute file edits to 60-minute multi-step system configurations — with per-task timeouts.

1.2 Agent Architecture

Our solver agent (i.e., Coding agent) employs two agent frameworks: (1) Strands Agents + AgentCore; (2) A standard ReAct loop + Sandbox/Docker. The solver has access to core tools:

• bash(): Execute shell commands in the Docker container (60s timeout per call)
• python(): Execute Python code in an isolated interpreter
• submit("DONE"): Signal task completion

The system prompt follows an Analysis/Plan format that structures the agent’s reasoning before each tool call.

1.3 The Evolution Framework (A-Evolve)

A-Evolve deploys a trajectory-only in-situation evolution loop as follows:

Solver Agent (bash/python/submit tools)
    ↓ solves tasks in Docker containers
Trajectory Observer (JSONL logs)
    ↓ records tool calls, outputs, ...
LLM Judge (scores 0-10 without ground-truth)
    ↓ evaluates trajectories/identify potential failure patterns/...
Evolver Agent (LLM agent accessing bash tool to modify the agent harness)
    ↓ mutates prompts/skills/tools/memories based on observations
Skill Library (lazy-loading)/Prompt/Tool Registry/Memory
    ↓ agent reads skills on demand/loads updated prompts/...
Next Batch's Solver

The key design constraint is that evolution operates without ground-truth feedback. The evolver never sees test results — it must infer what went wrong from the trajectory alone. This makes the problem significantly harder than test-guided optimization.

1.4 Experiment Metrics

For solver agent, we measure: (1) pass rate – resolved/total (%); (2) AVG turns; (3) AVG tool calls; (4) AVG used of skills; (5) Running time. For evolver agent, we measure: (1) number of created skills/tools/memories; (2) Changed prompt/memory chars.

1.5 Experiment Notation

We conduct our experiments in a principled way to systematically build a self-evolving coding agent. Particularly, we design the following chronologically thematic groups. Each group is labeled with a letter prefix, and individual experiments within a group are numbered sequentially:

1.6 Infrastructure: API reliability and context management

Our agent infrastructure addresses two key challenges: API reliability and context management. For API reliability, running 20 concurrent AWS Bedrock API calls requires robust error handling — we implement transient error retries (covering read timeouts, content filtering false positives, and connection hangs), timeout clock pausing during retry waits so that idle retry time does not consume the task’s budget, and a wall-clock safety cap to prevent infinite retry loops on persistent outages. For context management, we employ a summarization-based approach: when the conversation reaches the context window limit, we summarize the preceding conversation history into a condensed representation and continue from that summary, preserving critical state while freeing context capacity for continued reasoning.

2. Establishing the Baseline (Group A)

2.1 Opus 4.5 Baseline (A1)

Our initial experiments used Claude Opus 4.5. The vanilla ReAct coding agent scored AVG 60.15% across 5 runs (For simplicity, by default, all results are reported as AVG in 5 runs while some use 3 runs due to the API quota limit).

2.2 Opus 4.6 Baseline (A2)

When Claude Opus 4.6 was released (February 5, 2026), we used the same ReAct workflow and observed a significant capability jump:

Finding: Opus 4.6 solves tasks with 38.01% fewer input tokens and 16.70% fewer turns, while achieving a higher pass rate. The model is substantially more efficient and capable. In particular, we find Opus 4.6 will automatically self-verify the results.

3. Ceiling Analysis and Task Classification (Group B)

Before attempting to improve performance, we needed to (1) estimate the upper bound of the performance by computing pass@5 from A2 (B1a), increasing the per-task running time limit (B1b), and providing the environment signal in the ReAct loop through comparing the generated artifact and the ground-truth results (B1c). (2) understand the benchmark’s structure: which tasks are deterministically solvable, which are beyond the model’s capability, and which are stochastic. Then, we focus on these stochastic flaky tasks for improvement.

3.1 Ceiling Analysis (B1)

Finding: From B1a, we obtained the north star of Terminal-Bench 2.0 to guide our exploration (i.e., 85.88%). Meanwhile, we found providing environment signals does not help much in the coding tasks from B1c. From B1b, increasing the running time can help by ~3pp, indicating TB2 is a resource-heavy benchmark for evaluation.

3.2 Task Stability Classification (B2)

Across 5 independent Opus 4.6 runs, we classified all 89 tasks by their pass rate:

Finding for the theoretical ceiling: If all flaky tasks could be stabilized, the score would reach 85.88%. The 16 always-fail tasks represent a hard capability boundary.

3.3 Failure Pattern Taxonomy (B3)

Deep trajectory analysis — comparing successful and failed runs of the same task side by side — revealed distinct failure patterns as Finding:

These failure patterns help guide our next-stage experiment design for harness improvement (i.e., workflow, prompt, memory, tool, and skill).

4. Exploring Improvement Directions (Group C)

With the baseline, north star, and flaky tasks identified, we systematically explored four workspace components: tools, memory, system prompt, and skills.

4.1 System Prompt: Behavioral Strategy Evolution (C1)

We restricted the evolver to modifying only the system prompt to address the failure cases in B3, e.g., adding behavioral strategy rules like “Comprehensively understand the task and think various directions before starting to solve the problem.”.

Finding: C1a scored identically to vanilla A2 while longer and complex prompts hurt in C1b. The strategy rules are qualitatively sensible but quantitatively inert — Opus 4.6 already implements these strategies implicitly.

4.2 Memory: Episodic Learning Across Tasks (C2)

We tested memory injection — episodic observations from previous tasks used in future tasks by full loading. Memories accumulated across evolution cycles without providing task-relevant information.

Finding: Memory injection was consistently neutral or slightly negative. The evolver stored generic process observations (“always submit your answer,” “save intermediate results”) — advice the model already knows.

4.3 Tools: Universal vs Task-Specific (C3)

We tested the impact of adding pre-defined tools and self-generated tools. The pre-defined tool set is Bash, Python, and Submit. In the self-evolving stage, the evolver agent generates task-specific tools from the task trajectories.

Finding: (1) Submit tool helps because agent does not know when to stop. In the trajectory in C3a, we found agent sometimes continus solving the problem without submitting the results; (2) Self-generated task-specific tools hurt in the terminal/coding tasks. The similar coding tasks have been heavily trained in post-training stages only using universal tools (e.g., grep, bash). (3) The verification logic needs to be richer than what a tool interface naturally supports. The agent often called verify() but performed superficial checks.

4.4 Skills: The Most Promising Direction (C4)

We examine the influence of self-generated skills during evolutions. Skills serve as the domain-specific knowledge and provide guidance for agents when they solve similar tasks. These skills are used via lazy loading.

Finding: (1) In C4a, after evolver generated 3 seed skills, the agent chose to read at least one skill on 32.94% of tasks in the remaining tasks. (2) Even with more skills in C4b, we still witness consistent gain from skills. This indicates that skills delivered a measurable improvement when the skills were well-targeted, unlike tools, memory, or prompt changes.

4.5 Auto-Harness: The combination of prompt, tool, memory, and skill (C5)

Furthermore, we test the combination of agent harness. Based on the experiments C1-C4, we know skills help most and the lazy-loading mode will not hurt the performance much. On top of the skill, we explore different harness combination for deep-dive.

Finding: From the series of experiments in C5, we found that skills+{Prompt, tool, memory} hurts except C5d. This indicates that for the terminal bench 2.0, we should focus on the skill evolution for the north star shown in B1 experiment.

5. Skill Evolution Strategies (Group D)

For skill, it employ the lazy loading strategy. Particularly, we only list only skill names and one-line descriptions in the system prompt. The agent calls read_skill(name) to load full content on demand.

5.1 Skill Size Optimization (D1)

We examine the impact of skill size:

Finding: Skill size exhibits a clear sweet spot at 1,500–2,000 characters (D1d, 71.23%). Skills shorter than 1,000 characters lack sufficient domain detail to be useful, while skills exceeding 2,500 characters dilute the signal with redundant content. The drop-off is roughly symmetric: -4.02pp below the sweet spot (D1a) and -4.04pp above it (D1g). This guides our skill creation constraints in subsequent experiments.

5.2 Skill count budget and Per-Task Keyword Selection (D2)

We investigate the number of added skills. Besides, we also have our mitigation: match skill keywords against the task description, only inject skills with top-k relevance score ranked by N-gram overlapping.

Finding: A moderate skill budget of 6 (D2b) achieves the best score at 72.21%, outperforming both the tighter budget of 3 (D2a, 70.58%, with ratio of tasks using skills as 32.94%) and the unbounded setting (D2c, 68.23%). Unbounded skill injection degrades performance because the agent loads irrelevant skills that distract from the task. Per-task keyword selection (D2d, 71.94%) partially mitigates the unbounded problem by filtering skills for relevance, recovering most of the gain without a hard count limit.

5.3 Solver-Proposed Skills (D3)

We tested having the solver (with full environment context) propose skill drafts after each task, with the evolver judging the skills and curating proposals.

Finding: “Solver creates, evolver refines” (D3b) has the lowest score. The solver excels at extracting task-specific knowledge but fails at generalization. The evolver merged hyper-specific drafts into existing skills, replacing useful content with narrow tips like “ICS calendar parsing” and “OCR configuration.” “Solver creates, evolver generalizes” (D3c) partially recovered by instructing the evolver to “never shrink” existing skills and to generalize across them. However, the best result still comes from the evolver creating skills directly from trajectories (D3a). Lesson: The bottleneck is not knowledge extraction but knowledge generalization — the solver cannot reliably identify what transfers to unseen tasks.

5.4 Case study: The Process-Advice Trap (D4)

When we analyzed the long-horizon trajectory and read the skill content in D1-D3 experiments, we found that the evolver gravitates toward generic process advice. Without constraints, it produces skills like “always check if packages are installed” and “handle timeouts by trying alternatives.” We introduced a FORBIDDEN/REQUIRED blocklist as follows:

FORBIDDEN: Timeout handling, pip tips, session persistence, generic debugging
REQUIRED: Domain-specific libraries, verification steps, domain pitfalls ONLY

Finding: Negative prompting (D4b, blocking bad outputs) was more effective than positive steering (D4c). The combination (D4d) achieves the highest score but only marginally above negative prompting alone (+0.04pp), suggesting most of the gain comes from the blocklist. Without the blocklist, skills accumulate generic process advice; with it, they stay focused on domain-specific knowledge.

5.5 Case Study: Skill Evolution Non-Determinism (D5)

Similar to the non-determinism in LLM response generation and agent execution, we examine the the non-determinism in skill evolution. The five evolution runs from the identical seed. The experiment is shown below.

Finding: Five evolution runs from identical seeds produced substantially different skill content — each evolver wrote a different “textbook” based on which tasks happened to fail in each batch. Despite the content divergence, scores ranged from 69.65% to 74.77%, indicating that multiple distinct skill sets can be effective. This implies that evolved skills are samples from a large space of possibly-useful knowledge, heavily path-dependent on batch ordering and stochastic task outcomes. Ensembling diverse skill sets may be a promising future direction.

6. Self-Verification: Breaking the Ceiling (Group E)

For motivation, our error analysis (Section 3.3) revealed that many flaky task failures stem from unchecked assumptions rather than missing knowledge. The agent solves the task correctly most of the time but occasionally submits without verifying a critical decision — wrong normalization, broken package manager, overfitting to the example.

6.1 Ablation Study: Various Verification Approaches (E1)

We tested five verification delivery mechanisms on the 21 flaky tasks (Opus 4.6, no evolution):

The self-verification skill (E1d) was the clear winner. Critically, the combination of skill + prompt (E1f) performed worse than either alone — a counterintuitive result with an important explanation.

Why the Combination Hurts

The skill works because it’s lazy-loaded: the model reads it only when uncertain. The prompt instruction forces verification on every task, including those where the model is already confident. When both are present:

1. The prompt forces a verification step the model wouldn’t naturally take
2. The model also loads the skill (because it’s now in verification mode)
3. Redundant double-verification wastes 2-3 turns per task
4. The aggregate overhead costs more tasks than extra checking saves

Lesson: Let the model self-select when to verify rather than forcing it universally.

6.2 Scaling to Full Benchmark: The Initial Disappointment (E2)

The ablation showed the gain of self-verification skill on 21 flaky tasks. Then, we ran the same configuration on all 89 tasks with various self-verification skills.

Finding: Without any predefined skills (E2a), the baseline already achieves 72.21%. Adding a generic self-verification skill (E2b) actually hurts by 2.05pp — the skill contained advice like “check edge cases,” “test empty input,” “verify permissions,” which is knowledge the model already has (the same process-advice trap from Section 5.4). Only when we curated the skill with targeted assumption-checking content (E2c) did verification deliver a +2.06pp gain. This confirms that what the skill says matters far more than that a skill exists.

Guided by failure pattern analysis (Section 3.3), we replaced the generic advice with targeted assumption-checking:

Before (generic):

## 3. Edge Cases
- Check with empty input, special characters, large input
- Verify error handling works
- Check permissions (chmod, ownership)

After (targeted):

## 3. Check Your Assumptions
- If you chose between multiple approaches (e.g., normalization methods),
  verify your choice matches what the task/test expects
- If you installed or modified system packages, verify the package manager
  still works
- If your solution works on the provided example, consider whether it
  generalizes to different inputs

Each bullet addresses a real failure mode:

• Normalization choice: pytorch-model-cli failure (chose MNIST mean/std instead of raw /255)
• Package manager state: dna-assembly failure (apt-get timeout corrupted dpkg)
• Generalization: video-processing failure (MOG2 overfit to example video)

We also enhanced the server verification: “curl or wget to test endpoints — verify response content, not just that the port is open. Checking ports with ss or netstat is NOT enough.” This addresses install-windows-3.11 where the agent verified port 80 was open but never tested nginx content.

6.3 The Nudge: Directing the Model to Read the Skill (E3)

With improved content, we added a single line to the system prompt: “After you think you have completed the task, read the self-verification skill to verify your solution.”

Finding: The system prompt nudge dramatically increases skill loading frequency — from 24.71% (E3a) to 77.65%–90.59% (E3b/E3c). However, loading frequency alone does not guarantee improvement: E3b loads the skill on 77.65% of tasks yet drops performance by 6.03pp because the generic content wastes turns on checks the model already performs. E3c, with targeted content and 90.59% loading, achieves the best result at 76.47%. Lesson: Content quality matters more than loading frequency. The improvement comes from making each verification round productive (catches real bugs) rather than wasteful (checks things the model already knows).

7. Multi-Agent System Evolver (Group F)

Our single-agent evolver (used throughout Groups C–E) handles analysis, skill authoring, and quality control within one context window. This creates path dependence (D5) and forces a single agent to context-switch between fundamentally different cognitive tasks. We decomposed skill evolution into four specialized LLM agents — Orchestrator, Analyst, Author, and Critic — called MAS-Evolver.

MAS-Evolver uses an orchestrator-dispatch pattern on the Strands framework:

Key design decisions: (1) subagents are pure-reasoning with no tools — only the orchestrator mutates the workspace; (2) strict information boundaries prevent bias: the Analyst never sees existing skills, the Author never sees raw trajectories, and the Critic never sees the Author’s reasoning.

7.1 MAS Architecture Comparison (F1)

We compared three multi-agent coordination patterns against vanilla. The orchestrator pattern uses a central coordinator that dynamically sequences subagent calls with an iterative author–critic revision loop (up to 2 rounds per skill). The DAG pattern arranges agents in a fixed directed acyclic graph (Analyst → Author → Critic) with no revision cycles. The swarm pattern allows agents to self-organize without centralized control.

Finding: The orchestrator pattern (F1b, 69.28%) outperforms both DAG (F1c, -1.97pp) and swarm (F1d, -0.83pp). The DAG’s fixed pipeline cannot revise rejected skills — once the Critic rejects, the skill is lost. The swarm lacks centralized quality control, producing more skills but with inconsistent quality. The orchestrator’s iterative author–critic loop is the key differentiator: in Cycle 1, MAS-Evolver produced new skills (e.g., numerical-algorithm-implementation — covering log-space arithmetic in R, NaN/Inf guards, and adaptive rejection sampling.). The Critic approved it after one revision round. Cycles 2–4 repeated the process, creating additional skills until the budget was met.

7.2 Context Management in MAS (F2)

With the orchestrator architecture fixed from F1, we investigated how trajectory compression affects skill quality since the context management and agent communication is the key in MAS. Each evolution cycle processes batch trajectories that can be compressed for subagent. We compared three strategies: adaptive compression that adjusts granularity based on content complexity and remaining budget (e.g., context window limit by LLM model); whole-trajectory compression that keeps the task description intact and uniformly compresses the entire trajectory; and sliding-window compression that retains the task description and compresses only within a fixed recent window.

Finding: Adaptive compression (F2a, 72.94%) outperforms both whole-trajectory (F2b, -2.63pp) and sliding-window (F2c, -4.03pp) compression. Whole-trajectory compression discards too much detail uniformly — critical failure signals (e.g., a wrong normalization choice at turn 12) get compressed away alongside irrelevant boilerplate. Sliding-window compression is worse because it systematically drops early-trajectory context where task understanding and initial approach decisions occur, which are precisely the signals the Analyst needs. Adaptive compression preserves high-information-density segments (errors, decision points, tool output anomalies) while compressing routine segments (package installs, file reads) when the context window limit is reached. Overall, MAS-Evolver with adaptive compression achieves +7.03pp over vanilla, comparable to the single-agent skill evolution baseline (D2b: 72.21%).

8. Results Summary

8.1 Full Score Progression

Below we present all experiments organized by group. Detailed analysis for each experiment is in its respective section.

Group A: Baselines (Section 2)

Group B: Ceiling Analysis (Section 3)

Group C: Harness Component Exploration (Section 4)

Group D: Skill Evolution Strategies (Section 5)

Group E: Self-Verification (Section 6)

Group F: Multi-Agent System Evolver (Section 7)

Key Progression Summary

The path from baseline to best result:

8.2 Leaderboard

9. Key Learnings

What Works

1. Lazy skill loading. Let the model decide which skills to read. Without nudging, the model self-selects on ~25–33% of tasks (E3a, C4a); with a targeted nudge, up to 91% (E3c). System prompt stays at ~1,900 characters instead of 15,000+.

2. Targeted self-verification. A “Check Your Assumptions” wording in self-verification skill that verifies decision choices (normalization method, algorithm selection, package state) rather than generic edge cases. Content quality drives the improvement — generic verification hurts (E2b, -2.05pp) while targeted verification helps (E3c, +10.56pp).

3. FORBIDDEN blocklist. Blocking generic process advice forces domain-specific skill creation (D4b, +0.96pp over unblocked D4a). Negative prompting outperforms positive steering (D4b vs D4c).

4. Minimal prompt. The 1,368-character base prompt is near-optimal (C1b shows longer prompts hurt). Total budget including nudges: ~1,485 characters. Every addition must be weighed against attention dilution.

What Doesn’t Work

1. Solver-proposed skills. The solver extracts task-specific knowledge but fails at generalization (D3b: -4.86pp vs D3a).

2. Meta-skills. A “systematic-exploration” skill was loaded on only 1.18% of tasks. The model does not self-assess when it is stuck.

3. Unbounded skill injection. Removing skill count budgets degrades performance (D2c: 68.23% vs D2b: 72.21%) even with lazy loading, because the agent loads irrelevant skills.

Design Insights

1. Error analysis drives improvement. The +9.41pp gain came from trajectory analysis → failure pattern identification → targeted skill content.

2. Simple scaffold wins. Best configuration: minimal prompt (1.5K) + 5 focused skills (8.5K) + lazy loading + no evolution.

3. Multi-agent decomposition validates but doesn’t yet surpass. MAS-Evolver (72.94%) matches the single-agent baseline (D2b: 72.21%) while producing adversarially reviewed skills. The quality gate works — skills pass Critic review — but the architecture has not yet incorporated the refinements (blocklist, targeted verification) that drive the single-agent system to 76.47%.

10. Future Directions

A-Evolve+: Richer signal beyond in-situation evolution. The trajectory-only evolver sees compressed logs without pass/fail. Richer feedback (e.g., the pass/fail result, verifiable feedback) could improve skill quality.