How Good Is GLM5.2: Create Your Own Benchmark and Revisit Your Training Recipe

Bing He, Rui Sun, Zhan Shi, Hanqing Lu, Yisi Sang, Yuhao Zhang, Hao Zhou

Update: Upon GLM5.2’s release, we used the BF16 version for benchmark evaluation. Configuration details are in the Appendix.

1. Summary

We present a systematic evaluation of open-source LLM models on a self-created vertical-domain agentic multi-turn coding benchmark. The benchmark is fully self-owned with verifiable ground-truth answers, testing LLMs’ ability to solve complex problems through iterative tool use (Python, Bash, SQL, local-host KB search) within self-contained Docker environments. A ReAct-based agent loop serves as the evaluation harness (consistent with our ICLR’2026 TRAJECT-Bench framework), producing pass@1 and pass@5 accuracy scores for rigorous cross-model comparison.

Our evaluation yields three main findings:

1. Benchmark validation. We evaluate four models spanning 35B to 744B total parameters (Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, GLM5.2-744B-A40B). Accuracy scales monotonically from 0.173 to 0.456 with increasing active parameters, confirming the benchmark’s validity and discriminative power.

2. GLM5.2-744B-A40B performance. With 0.456 pass@1 and 0.592 pass@5, GLM5.2 achieves the highest accuracy among all evaluated open-source models, demonstrating strong agentic multi-turn coding capability with 40B active parameters.

3. Training recipe insights. Cross-referencing model behavior with our own RL experiments reveals: (a) PPO-trained models exhibit more exploratory multi-turn behavior than GRPO/DAPO/GSPO-trained models, consistent with GLM5.2’s PPO training recipe and pattern; (b) small models (35B) suffer severe catastrophic forgetting under aggressive RL despite strong task accuracy; (c) larger models (122B/397B) tolerate RL with only ~2% general capability degradation.

2. Introduction

2.1 Motivation: Why Create Your Own Benchmark?

Public benchmarks face two well-documented challenges: data contamination (models may have seen test data during training) and benchmark-maxing (developers optimize specifically for known benchmarks rather than general capability). Our prior work, TRAJECT-Bench (ICLR 2026), introduced trajectory-aware evaluation of LLM tool use with fine-grained diagnostics — tool selection correctness, argument accuracy, and dependency satisfaction. As a published benchmark, however, it is susceptible to leakage over time.

To obtain reliable, uncontaminated evaluation signals, we created a self-created vertical-domain agentic multi-turn coding benchmark with the following properties:

• Fully private and self-owned — never published, shared externally, or included in any training corpus, eliminating contamination risk entirely.
• Verifiable results — each task has a definitive ground-truth answer, enabling objective pass@1 and pass@5 accuracy measurement without reliance on model-based judges.
• Multi-turn agentic coding — models must autonomously decide which tools to invoke (Python, Bash, SQL, local-host KB search), write and execute code within Docker environments, interpret intermediate results, and iteratively build toward a solution through multiple rounds of tool use.
• Multi-domain grounding — tasks span three vertical domains (code migration, data analytics, financial QA) with realistic complexity that requires genuine multi-step reasoning.
• Comprehensive metrics — four complementary evaluation signals: accuracy, format compliance, CoT reasoning quality, and AVG turns for investigation efficiency.

2.2 Benchmark Overview

Task domains. The benchmark spans three vertical domains, each contributing 100 candidate tasks (300 total):

• Code migration — translating code between programming languages and verifying output equivalence
• Data analytics — multi-step numerical reasoning over structured databases and tables requiring query composition and interpretation
• Financial QA — quantitative reasoning over public companies’ latest earnings reports on the SEC website, involving multi-step calculations (ratios, growth rates, aggregations)

After manual quality review to ensure unambiguous ground-truth answers and appropriate difficulty calibration, 50 tasks were removed, yielding 250 tasks for downstream evaluation.

Task environment. Each task is instantiated in a self-contained Docker environment containing the necessary data, tables, and initial code artifacts. The model interacts with this environment through a set of tools:

• Python — write and run analytical code
• Bash — execute shell commands within the Docker container
• SQL query — retrieve structured data from database and table
• Search — query a local-host knowledge-base search engine for relevant information
• Submit — signal task completion with the final answer

Agent harness. The evaluation employs a ReAct-based agent loop (Reasoning + Acting) — the same framework used in our TRAJECT-Bench (ICLR 2026) paper. At each turn, the model observes the current state (previous tool outputs), reasons about what information is still needed, and selects the next tool action. The loop continues until the model submits a final answer or reaches the maximum turn limit.

Evaluation metrics:

• Accuracy (pass@1, pass@5): Did the model arrive at the correct answer? Measured as exact match against verifiable ground truth. pass@1 measures average accuracy across 5 independent samples; pass@5 measures the model’s ceiling (best-of-5).
• Format quality: Does the response follow structured output requirements across multi-turn episodes?
• CoT reasoning quality: Are the model’s cited evidence points and reasoning chains relevant to the ground-truth answer?
• Average turns: Number of tool-call rounds used (measures investigation depth and efficiency).

Scale: 250 tasks × 5 independent samples per task (1,250 episodes total).

2.3 Connection to TRAJECT-Bench (ICLR 2026)

TRAJECT-Bench (ICLR 2026) evaluated trajectory-level tool-use quality — whether tools are selected, parameterized, and ordered correctly — in controlled settings. This benchmark extends that philosophy to end-to-end problem solving: models must orchestrate sustained multi-turn tool use to solve complex problems and produce verifiable final answers. The key advance is verifiability — every task has a deterministic correct answer derivable from the data, enabling objective accuracy measurement without LLM-as-judge evaluation.

3. Benchmark Validation: Scaling Behavior

A well-designed benchmark should produce scores that scale predictably with model capability. We validate this by evaluating four models with increasing active parameters: Qwen3.5-35B-A3B (3B active), Qwen3.5-122B-A10B (10B), Qwen3.5-397B-A17B (17B), and GLM5.2-744B-A40B (40B). GLM5.2 is evaluated with its native thinking mode enabled.

3.1 Results

pass@1:

pass@5:

3.2 Analysis

1. Monotonic scaling confirms validity. Accuracy improves consistently with active parameters: 0.173 (3B) → 0.239 (10B) → 0.356 (17B) → 0.456 (40B), yielding approximately +8-12 points per 2× increase in active parameters. No floor effect (smallest model scores meaningfully) and no ceiling effect (largest model well below 1.0), confirming discrimination across a wide capability range.

2. Cross-family generalization. The scaling holds across two different model families (Qwen3.5 and GLM5.2), indicating the benchmark measures general agentic coding capability rather than family-specific artifacts.

3. Format quality correlates with scale. Structured output compliance improves from 0.425 to 0.927 — larger models better maintain instruction-following across long multi-turn episodes, a critical requirement for agentic deployment.

4. Investigation depth scales with capability. GLM5.2-744B-A40B uses the most turns (23.1) among all evaluated models, suggesting that more capable models naturally explore more deeply rather than guessing early. The benchmark rewards this thoroughness.

4. GLM5.2-744B-A40B: Detailed Performance Analysis

4.1 Performance Summary

4.2 Analysis

• Strong pass@5 performance. GLM5.2 achieves 0.592 pass@5, a +13.6 point improvement over pass@1 (0.456). This large gap indicates that GLM5.2’s iterative 23-turn investigation strategy produces correct answers at high quality, but with variance across independent samples — the model finds the right path frequently, just not always on the first try.

• Significant improvement over the next open-source model. Compared to Qwen3.5-397B-A17B (0.356 pass@1, 0.564 pass@5), GLM5.2 provides a +10.0 point improvement on pass@1 and +2.8 points on pass@5, consistent with the scaling behavior observed in Section 3.

• Thorough investigation strategy. GLM5.2 uses 23.1 turns on average — the highest among all evaluated models. This suggests an exploratory, hypothesis-testing approach where the model progressively refines its understanding through multiple tool calls rather than attempting to solve everything in a single step.

• Format quality at scale. With 0.927 format compliance, GLM5.2 reliably maintains structured output requirements across long multi-turn episodes (23+ turns), demonstrating strong instruction-following even during extended reasoning chains.

• Turn limit sensitivity. GLM5.2 requires a 50-turn limit because its thorough investigation style averages 23 turns. At 25 turns, 28% of samples hit step-exhaustion, reducing pass@1 by 8.6 points (0.370 → 0.456). Evaluation frameworks should accommodate diverse investigation strategies rather than imposing uniform step budgets that penalize thorough exploration.

5. Revisiting Training Recipes: What Benchmark Behavior Reveals

Beyond ranking models, the benchmark reveals behavioral patterns that carry direct implications for training. By cross-referencing tool-calling behavior with our own RL training experiments across three scales (35B, 122B, 397B), we identify actionable training recipe insights.

5.1 Turn-Count Behavior Across Training Approaches

A striking pattern emerges when comparing tool-calling behavior across different training methods:

5.2 Finding: PPO Encourages Exploration; GRPO/DAPO/GSPO Encourages Compression

In our RL training experiments across Qwen3.5-35B, 122B, and 397B:

• GRPO/DAPO/GSPO training consistently produces models that use fewer turns (2-7 depending on scale). The model learns a “mega-action” strategy — one comprehensive tool call that queries all relevant information simultaneously, followed by immediate answer generation.

• PPO training produces models that use more turns compared to GRPO/DAPO/GSPO at equivalent training steps. The model maintains a more exploratory, hypothesis-testing approach.

• GLM5.2 uses PPO (as stated in their technical report) and exhibits the most turns (23.1) among all evaluated models — consistent with PPO encouraging exploration through its value-function-based advantage estimation. Because PPO’s critic assigns value to intermediate states, the model receives positive learning signal for exploratory tool calls even when they don’t directly appear in the shortest path to the answer. In contrast, GRPO only rewards the final trajectory outcome, incentivizing the model to find the most compressed solution path.

This reveals a fundamental trade-off in RL algorithm choice for agentic tasks:

5.3 Finding: Scale Determines RL Stability and Catastrophic Forgetting

Our RL training experiments reveal a critical scale-dependent phenomenon:

At 35B scale (Qwen3.5-35B-A3B):

• RL training (GRPO/DAPO/GSPO) achieves strong benchmark accuracy (~0.50 pass@1)
• Model collapses to ~2 turns (single mega-action strategy)
• Severe catastrophic forgetting: reasoning, math, and instruction-following capabilities degrade significantly on general benchmarks
• The model “over-specializes” — gaining benchmark performance at the expense of general capability

At 122B and 397B scale:

• Similar turn-reduction occurs (base 14-17 turns → RL-trained 6-7 turns) but less extreme
• Catastrophic forgetting is dramatically reduced — general capabilities show only ~2% average degradation across held-out benchmarks
• The larger model has sufficient capacity to maintain general knowledge while learning specialized agentic behavior
• PPO training is more stable at these scales; at 35B, PPO tends to collapse or learn very slowly

Implication: Small models face a capacity bottleneck where RL training forces a trade-off between task performance and general capability. Larger models (122B+) can accommodate both without severe degradation — a critical finding for practitioners choosing model scale for RL post-training.

5.4 Recommendations for Training Recipe Design

Based on the combined evidence:

1. Algorithm selection determines exploration behavior. For tasks requiring thorough multi-step investigation, PPO’s exploration-encouraging property produces agents that investigate more deeply. For latency-sensitive deployment, GRPO/DAPO/GSPO produces more streamlined agents — but verify that general capabilities are preserved.

2. Scale determines safe RL regime. At ≤35B active parameters, aggressive RL risks severe catastrophic forgetting; conservative learning rates or capability-preservation regularization are necessary. At 122B+, models tolerate RL training with minimal degradation (~2%).

3. Turn count is a training diagnostic. Monitoring average turn count during RL training provides an early signal: sudden collapse to 2-3 turns indicates the model is learning shortcuts rather than genuine multi-step reasoning. This collapse correlates with catastrophic forgetting at small scale.

4. Step limits must match model strategy. GLM5.2 gained 8.6 points by increasing from 25 to 50 steps. Evaluation frameworks should accommodate diverse investigation strategies rather than imposing uniform step budgets that penalize thorough exploration.

6. Conclusion

We presented a systematic evaluation of open-source models on a self-created vertical-domain agentic multi-turn coding benchmark with verifiable results.

On benchmark design:

• The benchmark is valid and discriminative, showing consistent scaling from 35B to 744B active parameters across two model families without floor or ceiling effects.
• Private benchmarks with verifiable ground-truth answers provide uncontaminated evaluation signals that complement public benchmarks susceptible to leakage.

On model capability:

• GLM5.2-744B-A40B achieves 0.456 pass@1 and 0.592 pass@5, the highest among all evaluated open-source models. Its thorough 23-turn investigation strategy produces high-quality solutions with strong pass@5 ceiling performance.

On training recipes:

• PPO encourages exploration (more turns); GRPO/DAPO/GSPO encourages compression (fewer turns). Both achieve competitive accuracy through fundamentally different strategies.
• Catastrophic forgetting under RL is scale-dependent — severe at 35B, manageable (~2%) at 122B+.
• Turn-count collapse correlates with over-specialization — a diagnostic signal practitioners should monitor during RL training.

References

[1] Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding, Subhabrata Mukherjee, Suhang Wang, Yue Xing, Jiliang Tang, Benoit Dumoulin. “TRAJECT-Bench: A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use.” The International Conference on Learning Representations (ICLR), 2026.

[2] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. “ReAct: Synergizing Reasoning and Acting in Language Models.” arXiv preprint arXiv:2210.03629, 2022.

[3] Z.AI. “GLM-5.2: Built for Long-Horizon Tasks.” 2026-06-16. https://z.ai/blog/glm-5.2

[4] Qwen Team. “Qwen3.5: Towards Native Multimodal Agents.” 2026-02-15. https://qwen.ai/blog?id=qwen3.5

Appendix: Model Configurations

A.1 Qwen3.5 Configuration

All Qwen3.5 models (35B, 122B, 397B) are evaluated using the recommended thinking-mode settings:

A.2 GLM5.2-744B-A40B Configuration

GLM5.2 is configured as follows:

Note on GLM5.2 thinking mode: GLM5.2 produces structured responses with separate reasoning_content (internal chain-of-thought) and content (final answer) fields. The thinking process is always active and cannot be disabled.

Note on turn limit: GLM5.2 requires a 50-turn limit because its thorough investigation style averages 23 turns. At 25 turns, 28% of samples hit step-exhaustion, reducing pass@1 by 8.6 points (0.370 → 0.456).