How Good Are GLM5.2 and Kimi-K3: Create Your Own Benchmark and Revisit Your Training Recipe

Bing He, Rui Sun, Zhan Shi, Hanqing Lu, Yisi Sang, Yuhao Zhang, Hao Zhou

Update #2 (Jul 2026): Upon Kimi-K3’s release, we self-hosted the MXFP4 checkpoint and evaluated it on the identical 250 tasks with the identical 50-step limit — a clean 38.9-hour run with zero errors, zero truncations, and zero step-cap hits. This post is rewritten around the resulting three-family comparison.

Update #1 (Jun 2026): Upon GLM5.2’s release, we used the BF16 version for benchmark evaluation. Configuration details are in the Appendix.

1. Summary

We present a systematic evaluation of open-source LLM models on a self-created vertical-domain agentic multi-turn coding benchmark. The benchmark is fully self-owned with verifiable ground-truth answers, testing LLMs’ ability to solve complex problems through iterative tool use (Python, Bash, SQL, local-host KB search) within self-contained Docker environments. A ReAct-based agent loop serves as the evaluation harness (consistent with our ICLR’2026 TRAJECT-Bench framework), producing pass@1 and pass@5 accuracy scores for rigorous cross-model comparison. We evaluate three open-source model families: Qwen3.5 (35B, 122B, 397B), GLM5.2-744B-A40B, and Kimi-K3 (2.8T total / 104B active).

Our evaluation yields four main findings:

1. Benchmark validation across three families. Accuracy scales monotonically with active parameters across all five models — 0.173 (3B active) → 0.239 (10B) → 0.356 (17B) → 0.456 (40B) → 0.492 (104B) — confirming the benchmark’s validity and discriminative power with no floor or ceiling effect.

2. Kimi-K3 is the new open-source leader. With 0.492 pass@1 and 0.632 pass@5, K3 tops every open-source model we evaluated, posts the highest format quality we have recorded (0.997), and never once exhausted its 50-step budget.

3. The headline finding is the turn gap, not the accuracy gap. K3 averages 7.0 turns against GLM5.2’s 23.1 — the two turn distributions do not overlap. Once GLM5.2’s step-exhausted episodes are excluded, the two models are statistically identical on per-answer accuracy; K3’s real advantage is reaching the same answers at 3.3× lower inference cost, with materially better evidence citation.

4. Training recipe insights, extended. The two models fail differently — K3 by confident error, GLM5.2 by exhaustion — which demands different mitigations. Efficient convergence, not extended exploration, is what frontier accuracy looks like on this benchmark; more reasoning and more turns anti-correlate with accuracy for both models. Our own RL experiments (35B/122B/397B) continue to show that PPO encourages exploration, GRPO/DAPO/GSPO encourage compression, and catastrophic forgetting is scale-dependent.

2. Introduction

2.1 Motivation: Why Create Your Own Benchmark?

Public benchmarks face two well-documented challenges: data contamination (models may have seen test data during training) and benchmark-maxing (developers optimize specifically for known benchmarks rather than general capability). Our prior work, TRAJECT-Bench (ICLR 2026), introduced trajectory-aware evaluation of LLM tool use with fine-grained diagnostics — tool selection correctness, argument accuracy, and dependency satisfaction. As a published benchmark, however, it is susceptible to leakage over time.

To obtain reliable, uncontaminated evaluation signals, we created a self-created vertical-domain agentic multi-turn coding benchmark with the following properties:

• Fully private and self-owned — never published, shared externally, or included in any training corpus, eliminating contamination risk entirely.
• Verifiable results — each task has a definitive ground-truth answer, enabling objective pass@1 and pass@5 accuracy measurement without reliance on model-based judges.
• Multi-turn agentic coding — models must autonomously decide which tools to invoke (Python, Bash, SQL, local-host KB search), write and execute code within Docker environments, interpret intermediate results, and iteratively build toward a solution through multiple rounds of tool use.
• Multi-domain grounding — tasks span three vertical domains (code migration, data analytics, financial QA) with realistic complexity that requires genuine multi-step reasoning.
• Comprehensive metrics — four complementary evaluation signals: accuracy, format compliance, CoT reasoning quality, and AVG turns for investigation efficiency.

2.2 Benchmark Overview

Task domains. The benchmark spans three vertical domains, each contributing 100 candidate tasks (300 total):

• Code migration — translating code between programming languages and verifying output equivalence
• Data analytics — multi-step numerical reasoning over structured databases and tables requiring query composition and interpretation
• Financial QA — quantitative reasoning over public companies’ latest earnings reports on the SEC website, involving multi-step calculations (ratios, growth rates, aggregations)

After manual quality review to ensure unambiguous ground-truth answers and appropriate difficulty calibration, 50 tasks were removed, yielding 250 tasks for downstream evaluation.

Task environment. Each task is instantiated in a self-contained Docker environment containing the necessary data, tables, and initial code artifacts. The model interacts with this environment through a set of tools:

• Python — write and run analytical code
• Bash — execute shell commands within the Docker container
• SQL query — retrieve structured data from database and table
• Search — query a local-host knowledge-base search engine for relevant information
• Submit — signal task completion with the final answer

Agent harness. The evaluation employs a ReAct-based agent loop (Reasoning + Acting) — the same framework used in our TRAJECT-Bench (ICLR 2026) paper. At each turn, the model observes the current state (previous tool outputs), reasons about what information is still needed, and selects the next tool action. The loop continues until the model submits a final answer or reaches the maximum turn limit (50 steps for all models).

Evaluation metrics:

• Accuracy (pass@1, pass@5): Did the model arrive at the correct answer? Measured as exact match against verifiable ground truth. pass@1 measures average accuracy across independent samples; pass@5 measures the model’s ceiling (best-of-N).
• Format quality: Does the response follow structured output requirements across multi-turn episodes?
• CoT reasoning quality: Are the model’s cited evidence points and reasoning chains relevant to the ground-truth answer?
• Average turns: Number of tool-call rounds used (measures investigation depth and efficiency).

Scale: 250 tasks × 5 independent samples per task (1,250 episodes total).

2.3 Connection to TRAJECT-Bench (ICLR 2026)

TRAJECT-Bench (ICLR 2026) evaluated trajectory-level tool-use quality — whether tools are selected, parameterized, and ordered correctly — in controlled settings. This benchmark extends that philosophy to end-to-end problem solving: models must orchestrate sustained multi-turn tool use to solve complex problems and produce verifiable final answers. The key advance is verifiability — every task has a deterministic correct answer derivable from the data, enabling objective accuracy measurement without LLM-as-judge evaluation.

3. Benchmark Validation: Scaling Behavior Across Three Families

A well-designed benchmark should produce scores that scale predictably with model capability. We validate this with five models spanning three families and a 35× range of active parameters: Qwen3.5-35B-A3B (3B active), Qwen3.5-122B-A10B (10B), Qwen3.5-397B-A17B (17B), GLM5.2-744B-A40B (40B), and Kimi-K3 (104B active). GLM5.2 and K3 are evaluated with their native thinking modes enabled (always-on for both).

3.1 Results

pass@1:

pass@5:

3.2 Analysis

1. Monotonic scaling now holds across three families. Accuracy improves consistently with active parameters — 0.173 → 0.239 → 0.356 → 0.456 → 0.492 — with no floor effect (the smallest model scores meaningfully) and no ceiling effect (the largest model is well below 1.0). The trend generalizing across Qwen3.5, GLM5.2, and Kimi-K3 indicates the benchmark measures general agentic capability rather than family-specific artifacts.

2. Format quality correlates with scale, and K3 sets the record. Structured output compliance rises from 0.425 to 0.997. K3’s 0.997 is the highest we have measured: 99.8% of its final answers parsed on the first attempt.

3. We revise an earlier claim: investigation depth does not scale with capability. When GLM5.2 led this leaderboard, its 23.1 turns suggested that more capable models naturally explore more deeply. Kimi-K3 refutes that reading — it is the most capable open-source model here and also one of the most economical (7.0 turns). Turn count reflects a model’s convergence style, not its capability.

4. Kimi-K3: Detailed Performance Analysis

4.1 Performance Summary

The run itself was clean: zero episode errors, zero truncated steps, and the deepest prompt used only 3.7% of K3’s 1M-token context window — no result is confounded by resource limits.

4.2 The pass@1 → pass@5 gap is a consistency problem, not a knowledge problem

K3’s best-of-5 accuracy (0.632) sits far above its per-attempt accuracy (0.492) — a +0.140 gap. Looking at agreement across the independent samples of each task explains it:

• K3 is unanimous on 71% of tasks. On those, extra sampling cannot change the outcome — and on about two-fifths of them it is confidently wrong.
• The +0.140 gain comes almost entirely from the 29% of tasks where K3 is inconsistent: sampling several times finds a correct answer often enough to add 14 points.

Deployment implication: K3’s knowledge covers ~63% of the benchmark, but per-attempt reliability delivers only 49%. A self-consistency or majority-vote wrapper would recover a meaningful part of the gap — and because of K3’s turn efficiency (Section 5), multi-sample voting on K3 costs roughly the same as a single GLM5.2 attempt.

5. Head-to-Head: Kimi-K3 vs GLM5.2

Both models ran the identical 250 tasks with the identical 50-step limit, so there is no task-selection confound. Paired per-task tests (each task serving as its own difficulty control, n=250):

All paired tests are significant at the 95% level. Per task, K3 wins 47, loses 36, and ties 167 of 250.

5.1 The critical control: step exhaustion explains most of the accuracy gap

GLM5.2 hit the 50-turn cap on 7.2% of its samples; K3 hit it on none. Nearly all (96%) of GLM5.2’s capped episodes produced no parseable answer at all — they are not wrong answers but absent ones. Excluding them:

On episodes GLM5.2 actually finishes, the two models are statistically indistinguishable on accuracy and format. Of the three headline gaps, only CoT quality (+0.092 after the control) survives as a genuine per-answer capability difference — K3 cites metrics, directions, and deltas matching the ground truth more often, making its answers more auditable. The honest mechanism behind the accuracy gap is “K3 converges; GLM5.2 sometimes doesn’t” — and finishing within budget is itself part of the capability being measured.

K3 reaches the same per-answer accuracy as GLM5.2 using 3.3× fewer turns, with materially better evidence citation, and without ever exhausting its step budget.

5.2 The turn gap: two models on different scales of behavior

This is the largest, most robust, and most operationally consequential difference in the comparison (t = −30.5), and the only headline gap untouched by the step-exhaustion control.

The distributions are disjoint: K3’s worst episode of the entire run (18 turns) is below GLM5.2’s median (21). K3 is fully converged by 20 turns — the point where GLM5.2 has finished under half of its episodes — and nearly 80% of K3’s episodes finish within 8 turns, a threshold GLM5.2 reaches on 5.5%.

Inference cost: 3.29× fewer turns overall, and 3.55× fewer turns per correct answer (14.3 vs 50.6) — the metric that prices useful output rather than activity.

5.3 Different failure modes require different fixes

GLM5.2 thrashes: it spends nearly nine extra turns on episodes it ultimately gets wrong, and sometimes never concludes. K3 commits to an answer at roughly the same cost whether or not it is right. K3’s failure mode is confident error; GLM5.2’s is exhaustion. The mitigations differ accordingly — calibration or self-consistency voting for K3, a larger step budget for GLM5.2.

6. What the Benchmark Reveals About All Models

Several structural findings hold for both frontier open-source models, which makes them properties of the task rather than of any one model.

1. Tasks requiring a two-part answer are a shared ceiling. About 21% of the benchmark has a two-part ground truth, and both K3 and GLM5.2 score exactly 0.071 there — a collapse from their ~0.56–0.61 single-part accuracy. The mechanism, measured on K3, is omission rather than misidentification: it identifies the first part correctly 57.5% of the time but volunteers a second part in only 30% of the episodes that need one — while over-producing second parts on 20% of tasks that don’t. If K3 merely matched its own single-part accuracy on these tasks, its pass@1 would rise by +0.113. Because both models fail identically, this looks addressable through prompt and task design rather than model capability — the highest-value follow-up we found.

2. Partial understanding goes unrewarded by exact match. Roughly a quarter of all samples identify the main answer correctly yet score zero on exact match because a component is missing or wrong. Counting those, K3 “understands” about 75% of episodes against a 49.2% exact-match headline.

3. A 26× accuracy spread across task categories. Task categories with a discrete, single-signal signature approach 0.88 accuracy and finish fastest; categories requiring diffuse, multi-signal, or counterfactual reasoning fall as low as 0.034 and consume the most turns. Effort and results are inversely related across categories — more work, worse outcomes.

4. More compute does not buy more accuracy. K3’s accuracy falls monotonically as its reasoning grows — from 0.674 on its shortest-thinking episodes to 0.348 on its longest — and the same holds for turns (0.570 at ≤4 turns vs 0.348 at ≥9). Long episodes are a symptom of task difficulty, not a path to better answers. Raising step budgets or reasoning effort is not the lever here; neither model ever escalates effort on the two-part tasks it fails, confirming the failures are omissions rather than exhausted attempts.

7. Revisiting Training Recipes: What Benchmark Behavior Reveals

By cross-referencing the behavior of all three families with our own RL training experiments across three scales (35B, 122B, 397B), we extend the training recipe insights from the first version of this post.

7.1 Turn-Count Behavior Across Training Approaches

7.2 Finding: PPO Encourages Exploration; GRPO/DAPO/GSPO Encourages Compression

In our RL training experiments across Qwen3.5-35B, 122B, and 397B:

• GRPO/DAPO/GSPO training consistently produces models that use fewer turns (2-7 depending on scale). The model learns a “mega-action” strategy — one comprehensive tool call that gathers all relevant information at once, followed by immediate answer generation. Our further-iterated runs at the larger scales reach up to ~0.56 pass@1 in this compressed regime, the strongest accuracy we have obtained on this benchmark.

• PPO training produces models that use more turns at equivalent training steps, maintaining an exploratory, hypothesis-testing approach. GLM5.2 uses PPO (per its technical report) and exhibits the most turns (23.1) — consistent with PPO’s value-function-based advantage estimation crediting intermediate exploration, where GRPO’s trajectory-level reward incentivizes the shortest solution path.

7.3 Finding: Efficient Convergence, Not Exploration, Is What Frontier Accuracy Looks Like

Kimi-K3 adds a data point the original dichotomy could not supply: the top open-source score on this benchmark comes from a model that converges in 7 turns, never exhausts its budget, and barely varies its effort with task difficulty. Combined with the finding that accuracy anti-correlates with turns and reasoning length for both frontier models, the picture is:

• Exploration is a cost profile, not a capability signal. GLM5.2’s 23-turn style buys no accuracy over K3’s 7-turn style and produces a step-exhaustion failure mode (7.2% of samples) that K3 simply does not have.
• Turn-count collapse remains a diagnostic — but with a sign correction. In our 35B RL runs, collapse to 2-3 turns accompanied catastrophic forgetting and signaled shortcut learning. K3’s 7-turn behavior with zero cap-hits is the opposite: compressed and accurate. Turn count alone is ambiguous; pair it with cap-hit rate and accuracy before reading it as a health signal.
• Failure modes dictate the fix. Exhaustion-prone models (GLM5.2) benefit from bigger budgets — GLM5.2 gains 8.6 points going from 25 to 50 steps. Confident-error models (K3) do not: K3’s improvement path is calibration and self-consistency voting (+0.140 headroom), which its own efficiency makes affordable.

7.4 Finding: Scale Determines RL Stability and Catastrophic Forgetting

Unchanged from our earlier experiments, and worth restating:

• At 35B scale, aggressive RL (GRPO/DAPO/GSPO) achieves strong benchmark accuracy (~0.50) but collapses the model to ~2 turns and induces severe catastrophic forgetting — reasoning, math, and instruction-following degrade significantly on general benchmarks.
• At 122B and 397B scale, similar turn-compression occurs without the damage: general capabilities degrade only ~2% on held-out benchmarks, and PPO training is markedly more stable.

Implication: small models face a capacity bottleneck where RL trades general capability for task performance; 122B+ models can hold both.

7.5 Recommendations for Training Recipe Design

1. Pick the RL algorithm for the deployment cost profile. GRPO/DAPO/GSPO yields streamlined agents suited to latency-sensitive serving; PPO yields deeper investigators. Neither buys accuracy by itself at the frontier — verify general capabilities either way.
2. Monitor turn count together with cap-hit rate and accuracy. Collapse plus forgetting is a failure; compression plus zero cap-hits (the K3 profile) is the goal state.
3. Match step limits to the model’s convergence style. A uniform budget penalizes exploratory models (GLM5.2 loses 8.6 points at 25 steps) and wastes nothing on efficient ones (K3 never passed 18).
4. Buy consistency, not depth, at test time. For confident-error models, multi-sample voting converts ceiling into reliability; for K3 the 5-sample cost is comparable to one GLM5.2 attempt.

8. Serving Insights for Self-Hosting Open-Source Models

Evaluating these models means serving them, and several deployment lessons generalize:

1. Serving configuration can cost more than a model-generation gap. A GLM5.2 deployment served in FP8 with a 40K context window scored 0.341 pass@1 — 0.115 below the same model served in BF16 with its full context (0.456). Before comparing models, verify the deployment; the “same model” is not the same model under a degraded serving recipe.

2. Respect each model’s specified inference recipe. K3’s authors fix temperature at 1.0 / top_p 0.95 and expose reasoning_effort as the quality lever; lowering the temperature is off-recipe. GLM5.2’s recommended temperature is 0.6. This means cross-model comparisons are model-as-deployed comparisons, not controlled ablations at matched sampling — we state this plainly rather than pretending the knobs match.

3. Preserved-thinking modes fail silently when mishandled. K3 is trained in preserved-thinking mode: the harness must echo reasoning_content back on every assistant turn. With the ordinary OpenAI-style content-only echo, the model silently degrades with no error. Our run’s parser diagnostics fired zero times across all episodes, confirming the echo path worked — but this is an easy way to produce quietly wrong benchmark numbers.

4. Quantization may not be a choice. K3 ships only as an MXFP4 (QAT) checkpoint — BF16/FP8 weights do not exist. On H200 (no native MXFP4 tensor cores), the workable path is W4A16 dequantization-in-register, which costs throughput but adds no quantization beyond what the authors shipped.

5. Architecture dictates parallelism. GLM5.2’s data-parallel-attention recipe would be actively harmful for K3, whose hybrid design keeps a fixed recurrent state per request that grows 32× under that scheme. Parallelism strategies do not transfer across architectures; they follow from the memory model.

6. Speculative decoding is a speed lever, not a quality lever. Neither model used it here (verified on both), so neither had a draft-model advantage; enabling it later would cut wall-clock at unchanged accuracy, since speculation is output-equivalent by construction.

9. Conclusion

We presented a systematic evaluation of three open-source model families on a self-created vertical-domain agentic multi-turn benchmark with verifiable results.

On benchmark design:

• The benchmark is valid and discriminative across three families and a 35× active-parameter range, with monotonic scaling and no floor or ceiling effects.
• Private benchmarks with verifiable ground truth provide uncontaminated signals — and their diagnostic structure (two-part tasks, category spreads, partial-credit ladders) reveals model behavior that a single headline number hides.

On model capability:

• Kimi-K3 leads open-source at 0.492 pass@1 / 0.632 pass@5 with record format compliance (0.997), converging in 7.0 turns without ever exhausting its budget.
• The turn gap, not the accuracy gap, is the story. K3 matches GLM5.2’s per-answer accuracy at 3.3× fewer turns with materially better evidence citation. K3 fails by confident error, GLM5.2 by exhaustion — different problems with different fixes.
• The clearest improvement paths are structural: the two-part-answer omission (+0.113 pass@1 available, likely via prompting) and per-attempt consistency (+0.140 between pass@1 and pass@5, via voting that K3’s efficiency makes affordable).

On training recipes:

• PPO encourages exploration; GRPO/DAPO/GSPO encourages compression — but K3 shows that efficient convergence, not exploration, is what frontier accuracy looks like on this benchmark.
• Catastrophic forgetting under RL is scale-dependent — severe at 35B, manageable (~2%) at 122B+.
• Turn count is a diagnostic only in context: collapse with forgetting is shortcut learning; compression with zero cap-hits and high accuracy is the goal state.

References

[1] Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding, Subhabrata Mukherjee, Suhang Wang, Yue Xing, Jiliang Tang, Benoit Dumoulin. “TRAJECT-Bench: A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use.” The International Conference on Learning Representations (ICLR), 2026.

[2] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. “ReAct: Synergizing Reasoning and Acting in Language Models.” arXiv preprint arXiv:2210.03629, 2022.

[3] Z.AI. “GLM-5.2: Built for Long-Horizon Tasks.” 2026-06-16. https://z.ai/blog/glm-5.2

[4] Qwen Team. “Qwen3.5: Towards Native Multimodal Agents.” 2026-02-15. https://qwen.ai/blog?id=qwen3.5

[5] Moonshot AI. “Kimi-K3.” 2026-07. https://huggingface.co/moonshotai/Kimi-K3

Appendix: Model Configurations

A.1 Qwen3.5 Configuration

All Qwen3.5 models (35B, 122B, 397B) are evaluated using the recommended thinking-mode settings:

A.2 GLM5.2-744B-A40B Configuration

Note on turn limit: GLM5.2’s thorough investigation style averages 23 turns; at a 25-turn limit, 28% of samples hit step-exhaustion, reducing pass@1 by 8.6 points (0.370 → 0.456). The 50-turn limit is required for a fair reading of this model.

A.3 Kimi-K3 Configuration

Fairness note. Step limit, token limits, task set, and the no-speculative-decoding setting were matched with the GLM5.2 baseline. Temperature (1.0 vs 0.6) and precision (MXFP4 vs BF16) differ by necessity — they are properties of the models as shipped, making this a model-as-deployed comparison rather than a controlled ablation.