nanobot-runtime/results/2026-06-07_ollama-cloud-agent-model-comparison.md

Ollama Cloud Agent Model Comparison — Nanobot Deployment

Date: 2026-06-07
Baseline: glm-5.1:cloud
Scope: Evaluate all Ollama Cloud models against 12 criteria for sustained nanobot agent use.

---

Executive Summary

GLM-5.1:cloud remains the best default for nanobot agent deployment on Ollama Cloud. It offers the best balance of speed (~198 tok/s), proven agentic reliability, MIT license, 200K context, and no known language-drift or tool-calling blockers.

Viable alternatives (with tradeoffs):

• deepseek-v4-flash:cloud — if you need 1M context and can tolerate slower speed. MIT license, open weights.
• qwen3.5:397b-cloud — if you need multimodal + 1M context + explicit Czech support (201 languages). Apache 2.0. Reported as slow with accuracy issues on Ollama Cloud.
• devstral-2:123b-cloud — if the workload is purely coding-heavy and 128K context is sufficient. Strong SWE-Bench / Terminal-Bench scores. Apache 2.0.

Not recommended due to blockers:

• minimax-m3:cloud — critical tool-result message bug on Ollama Cloud (ollama/ollama #16389).
• kimi-k2.6:cloud — random Chinese output drift (critical risk for Czech use).
• deepseek-v4-pro:cloud — strongest benchmarks but 15.4 tok/s and 57s TTFT cold-start make it impractical for interactive agent work.

GLM-5.2 status: Not released. No official announcement from Z.AI as of June 2026.

---

Comparison Table

Model	Speed (tok/s)	TTFT	SWE-Bench V	SWE-Bench Pro	Terminal-Bench	MCP-Atlas	HLE	Code Arena Elo	Tool Reliability	Czech / Multilingual	Context	Multimodal	License	Verbosity	Known Bugs	Pricing (OpenRouter proxy)	Long-Horizon	Self-Host
glm-5.1:cloud	~198	Low	~58.4 (Pro)	58.4	63.5	71.8	52.3	1530	Excellent	No Czech claim; no drift observed	200K (198K Ollama)	No	MIT	Low	None known	~$4/M out	Proven	Yes
deepseek-v4-pro:cloud	~15.4	57s cold	80.6	—	67.9	74.2	56.2	—	Good	Strong multilingual (MMMLU 90.3)	1M	No	MIT	Medium	Extreme variance	$1.74/M in	Unknown	Yes
deepseek-v4-flash:cloud	~30-50*	Moderate	~75*	—	~60*	—	—	—	Good	Strong multilingual	1M	No	MIT	Medium	None known	$0.14/M in	Unknown	Yes
qwen3.5:397b-cloud	~10-20*	High	~66-70	—	~59.3	—	—	—	Good	201 languages incl. Czech	1M	Yes	Apache 2.0	Medium	"Too slow, accuracy issues" per user benchmark	—	Unknown	Yes
qwen3.5:cloud	~20-40*	Moderate	~60-65	—	~55	—	—	—	Good	201 languages	256K	Yes	Apache 2.0	Medium	None known	—	Unknown	Yes
minimax-m3:cloud	~40-60*	Low	—	—	—	—	—	—	Broken	Undeclared	512K	Yes	Open weights (pending)	—	Tool result messages fail (#16389)	$0.60/M in	N/A	Yes (pending)
kimi-k2.6:cloud	~30-50*	Moderate	80.2	—	66.7	—	54.0	—	Good	No Czech claim; random Chinese drift	256K	Yes	Modified MIT	Medium	Chinese output bug; OR context bug (32K)	$0.60/M in	200-300 tool calls	Yes
kimi-k2-thinking:cloud	~25-40*	Moderate	—	—	—	—	—	—	Good	No Czech claim	256K	No	Modified MIT	High	Older (Nov 2025)	—	200-300 seq tool calls	Yes
kimi-k2.5:cloud	~30-50*	Moderate	—	—	—	—	—	—	Good	No Czech claim	256K	Yes	Modified MIT	Medium	Older (Jan 2026)	—	Unknown	Yes
nemotron-3-ultra:cloud	~50-80*	Low	~60-79*	—	—	74.2	—	—	Unknown	Undeclared	200K	No	NVIDIA Open	Low	Too new (Jun 4 2026)	$0.60/M in	Unknown	Yes (NVFP4)
nemotron-3-super:cloud	~60-100*	Low	60.47	—	—	—	—	—	Unknown	Undeclared	1M	No	NVIDIA Open	Low	None known	—	Unknown	Yes
gemma4:31b-cloud	~80-120*	Low	~52.0	—	~29.2	—	—	—	Native FC	140+ languages	256K	Yes	Apache 2.0	Low	None known	—	Unknown	Yes
devstral-2:123b-cloud	~40-60*	Moderate	72.2	—	77.3	—	—	—	Good	Undeclared	128K	No	Apache 2.0	Medium	None known	—	Unknown	Yes
gpt-oss:120b-cloud	~30-50*	Moderate	~41.9	—	—	—	—	—	Good	Undeclared	128K	No	Apache 2.0	Medium	Older (Aug 2025)	$0.039/M in	Unknown	Yes
gemini-3-flash-preview:cloud	~60-80*	Low	—	—	—	—	—	—	Good	Strong (Google)	1M	Yes	Proprietary	Low	Older (Dec 2025)	—	Unknown	No
qwen3-coder-next:cloud	~40-60*	Moderate	~70.6	—	—	—	—	—	Good	201 languages	512K	No	Apache 2.0	Medium	None known	—	Unknown	Yes

*Speed estimates marked with * are inferred from similar-size MoE models or provider benchmarks, not direct Ollama Cloud measurements. GLM-5.1's ~198 tok/s is the only Ollama Cloud-specific speed figure found in the knowledge base.

---

Detailed Analysis by Criterion

1. Speed (TTFT, throughput, wall-clock latency)

• GLM-5.1 is the clear speed leader on Ollama Cloud at ~198 tok/s.
• DeepSeek V4-Pro is the slowest at ~15.4 tok/s with extreme variance and 57s cold-start TTFT.
• Gemma 4 31B and Nemotron 3 Super are likely the fastest among alternatives due to small active parameter counts (31B dense, 12B active MoE).
• Qwen3.5:397B is reported as "too slow" in user benchmarks.

2. Intelligence (Benchmarks)

• SWE-Bench Verified leaders: DeepSeek V4-Pro (80.6%), Kimi K2.6 (80.2%), Devstral 2 (72.2%), Qwen3.5-397B (~66-70%), Nemotron 3 Super (60.47%), Gemma 4 31B (~52%), GPT-OSS 120B (~41.9%).
• SWE-Bench Pro: GLM-5.1 leads open models at 58.4%.
• Terminal-Bench 2.0: DeepSeek V4-Pro (67.9%), Kimi K2.6 (66.7%), Devstral 2 (77.3% — highest reported), GLM-5.1 (63.5%).
• MCP-Atlas: Nemotron 3 Ultra (74.2%), DeepSeek V4-Pro (74.2%), GLM-5.1 (71.8%).
• HLE: Kimi K2.6 (54.0%), GLM-5.1 (52.3%).
• Code Arena Elo: GLM-5.1 at 1530 (#3 globally for agentic web dev).

3. Tool Calling Reliability & Schema Adherence

• GLM-5.1: 99.6% schema adherence (per prior research), no known tool-calling failures.
• MiniMax M3: Critical blocker — fails on tool result messages via Ollama Cloud OpenAI-compatible endpoint (issue #16389, 6 days old as of Jun 7). Returns empty responses.
• Kimi K2.6: Good tool-call reliability but known OpenRouter context-length bug (reports 32K instead of 256K) that may affect Ollama.
• Gemma 4 31B: Native function calling support.
• Nemotron 3 Ultra: Too new; no verified tool-calling data yet.

4. Czech / Multilingual Support & Language Drift Risk

• Qwen3.5 (all variants): Explicitly claims 201 languages including Czech. Best documented multilingual support.
• Gemma 4 31B: Claims 140+ languages, Apache 2.0.
• DeepSeek V4: Strong multilingual (MMMLU 90.3, C-Eval 93.1) but no explicit Czech claim.
• GLM-5.1: No explicit Czech claim, but no known language drift in practice.
• Kimi K2.6: Critical risk — multiple user reports of random Chinese output even with English prompts. No explicit Czech support claim.
• MiniMax M3: No explicit multilingual claim.
• Nemotron / Devstral / GPT-OSS: No explicit Czech claims.

5. Context Window Size

• 1M tokens: deepseek-v4-pro, deepseek-v4-flash, nemotron-3-super, qwen3.5:397b-cloud, gemini-3-flash-preview
• 512K: minimax-m3, qwen3-coder-next
• 256K: kimi-k2.6, kimi-k2-thinking, kimi-k2.5, qwen3.5:cloud, gemma4:31b-cloud, devstral-2
• 200K: glm-5.1, nemotron-3-ultra
• 128K: gpt-oss:120b-cloud

6. Multimodality

• Multimodal: minimax-m3, kimi-k2.6, kimi-k2.5, qwen3.5:397b-cloud, qwen3.5:cloud, gemma4:31b-cloud, gemini-3-flash-preview
• Text-only: glm-5.1, deepseek-v4-pro, deepseek-v4-flash, kimi-k2-thinking, qwen3-coder-next, devstral-2, gpt-oss, nemotron-3-super/ultra

7. Open Weights & License

• MIT: GLM-5.1, GLM-5, DeepSeek V4-Pro/Flash
• Apache 2.0: Qwen3.5/Qwen3.6/Qwen3-coder, Gemma 4, GPT-OSS 120B, Devstral 2
• Modified MIT: Kimi K2.6, Kimi K2-thinking, Kimi K2.5
• NVIDIA Open License: Nemotron 3 Ultra/Super
• Proprietary: Gemini-3-flash-preview
• MiniMax M3: Open weights promised ~10 days after launch (early June 2026) — likely available by now.

8. Verbosity (Tokens per Answer)

• Low: GLM-5.1 ("nejmenší verbosity z MoE rodiny"), Nemotron 3 Ultra/Super (up to 30% fewer tokens per turn), Gemma 4 31B
• Medium: DeepSeek V4, Qwen3.5, Devstral 2, Kimi K2.6
• High: Kimi K2-thinking (reasoning model)

9. Known Bugs / Blockers on Ollama Cloud

• MiniMax M3: Tool result message failures (#16389) — deploy blocker.
• Kimi K2.6: Random Chinese output drift — deploy blocker for Czech use.
• DeepSeek V4-Pro: Extreme latency variance, 57s cold-start TTFT — usability issue.
• Qwen3.5:397B: User-reported "too slow, accuracy issues" on Ollama Cloud.
• GLM-5.1: No known bugs.

10. Pricing (OpenRouter proxy — Ollama Cloud is flat-rate $20/mo Pro)

• Cheapest input: GPT-OSS 120B ($0.039/M), DeepSeek V4-Flash ($0.14/M)
• Mid-range: DeepSeek V4-Pro ($1.74/M), MiniMax M3 ($0.60/M), Kimi K2.6 ($0.60/M), Nemotron 3 Ultra ($0.60/M)
• Most expensive: GLM-5.1 (~$4/M output)
• Note: Ollama Cloud Pro is flat-rate $20/month; per-token pricing only matters if switching to API/OpenRouter fallback.

11. Long-Horizon Agent Stability (Multi-turn, hundreds of tool calls)

• GLM-5.1: Proven over "hundreds of rounds" — best sustained productivity per user experience.
• Kimi K2-thinking: Explicitly designed for 200-300 sequential tool calls.
• Kimi K2.6: Supports 200-300 sequential tool calls.
• Nemotron 3 Ultra: Marketed for "long-running agents" but too new for verification.
• DeepSeek V4: Unknown for sustained multi-turn agent use on Ollama Cloud.

12. Self-Host Fallback Possibility

• All models except Gemini-3-flash-preview have open weights available on Hugging Face.
• NVFP4 quantization: Nemotron 3 Ultra/Super require NVIDIA-specific formats.
• Hardware requirements:
  • GLM-5.1: ~198K context on Ollama; self-host requires significant VRAM.
  • DeepSeek V4-Flash: 284B total / 13B active — efficient MoE, viable on consumer hardware.
  • Qwen3.5:397B: 397B total / 17B active — large but efficient.
  • Gemma 4 31B: Dense 31B — fits on 24GB GPU.
  • Devstral 2 123B: Large but coding-optimized.

---

Recommendations

Primary Default (No Change)

glm-5.1:cloud remains the best default nanobot agent model on Ollama Cloud.

Why:

• Fastest measured speed (~198 tok/s)
• Best proven track record for sustained agent sessions
• MIT license
• No known bugs or language drift
• Strong benchmark suite (SWE-Bench Pro 58.4, Terminal-Bench 63.5, MCP-Atlas 71.8, Code Arena Elo 1530)
• Low verbosity = lower token burn

Alternative Tier 1 (Specific Needs)

1. deepseek-v4-flash:cloud — Choose if you need 1M context for large codebase analysis or long-document processing. MIT license, open weights, cheaper than Pro. Tradeoff: slower than GLM-5.1 (~30-50 tok/s estimated).
2. qwen3.5:397b-cloud — Choose if you need multimodal input (screenshots, diagrams) or explicit Czech language support (201 languages claimed). Apache 2.0, 1M context. Tradeoff: slow, reported accuracy issues on Ollama Cloud.

Alternative Tier 2 (Niche Use)

3. devstral-2:123b-cloud — Choose for pure coding-heavy workloads with strong benchmark scores (SWE-Bench 72.2%, Terminal-Bench 77.3%). Tradeoff: 128K context limit, no multimodal.
4. gemma4:31b-cloud — Choose if you need a fast, lightweight alternative with native function calling and 140+ language support. Tradeoff: weaker agent benchmarks (SWE-Bench ~52%, Terminal-Bench ~29%).

Avoid (Blockers)

• minimax-m3:cloud — Tool-calling bug makes it unusable for agent work until Ollama fixes #16389.
• kimi-k2.6:cloud — Chinese language drift is unacceptable for Czech-language agent use.
• deepseek-v4-pro:cloud — 15.4 tok/s and 57s TTFT make it impractical for interactive agent sessions despite top benchmarks.

Watch List

• nemotron-3-ultra:cloud — Too new (released June 4, 2026). Promising specs (550B/55B, 1M ctx, low verbosity) but needs real-world agent validation on Ollama Cloud.
• qwen3-coder-next:cloud — Strong coding focus, 512K context, Apache 2.0. Good candidate if coding is the primary workload.

---

GLM-5.2 Status

Not released. As of June 7, 2026, Z.AI has made no official announcement of GLM-5.2. Reddit speculation from April 2026 suggested 50-83 days from GLM-5.1 launch (April 7, 2026), implying a June-July 2026 window, but no confirmation exists. It is not available on Ollama Cloud.

---

Sources & Methodology

• Ollama Cloud model listings: ollama.com/search?c=cloud
• Benchmark aggregators: llm-stats.com, benchlm.ai, benchmark.space, swebench.com
• Vendor technical reports: NVIDIA Nemotron 3 Ultra (Jun 4, 2026), DeepSeek V4 (Apr 24, 2026), Qwen3.5/3.6 blog posts, Kimi K2.6 blog, Z.AI GLM-5.1 page
• Community benchmarks: ollama-cloud-benchmark GitHub (erikwangz), dev.to user benchmarks
• Bug trackers: ollama/ollama #16389 (MiniMax M3), Cursor/Reddit user reports (Kimi K2.6 Chinese drift)
• Pricing: OpenRouter proxy rates (Ollama Cloud itself is flat-rate $20/mo Pro)

Note on speed: Only GLM-5.1 has a direct Ollama Cloud speed measurement in our knowledge base (~198 tok/s). All other speed figures are estimates inferred from MoE active-parameter counts, provider benchmarks, or similar-platform measurements. Actual Ollama Cloud performance may vary due to load, cold starts, and quantization.