Zalohovani vsech podstatnych souboru
This commit is contained in:
lachtan
187
results/2026-06-07_ollama-cloud-agent-model-comparison.md
Normal file
187
results/2026-06-07_ollama-cloud-agent-model-comparison.md
Normal file
@@ -0,0 +1,187 @@
|
||||
# Ollama Cloud Agent Model Comparison — Nanobot Deployment
|
||||
|
||||
**Date:** 2026-06-07
|
||||
**Baseline:** `glm-5.1:cloud`
|
||||
**Scope:** Evaluate all Ollama Cloud models against 12 criteria for sustained nanobot agent use.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**GLM-5.1:cloud remains the best default** for nanobot agent deployment on Ollama Cloud. It offers the best balance of speed (~198 tok/s), proven agentic reliability, MIT license, 200K context, and no known language-drift or tool-calling blockers.
|
||||
|
||||
**Viable alternatives (with tradeoffs):**
|
||||
- **`deepseek-v4-flash:cloud`** — if you need 1M context and can tolerate slower speed. MIT license, open weights.
|
||||
- **`qwen3.5:397b-cloud`** — if you need multimodal + 1M context + explicit Czech support (201 languages). Apache 2.0. Reported as slow with accuracy issues on Ollama Cloud.
|
||||
- **`devstral-2:123b-cloud`** — if the workload is purely coding-heavy and 128K context is sufficient. Strong SWE-Bench / Terminal-Bench scores. Apache 2.0.
|
||||
|
||||
**Not recommended due to blockers:**
|
||||
- `minimax-m3:cloud` — critical tool-result message bug on Ollama Cloud (ollama/ollama #16389).
|
||||
- `kimi-k2.6:cloud` — random Chinese output drift (critical risk for Czech use).
|
||||
- `deepseek-v4-pro:cloud` — strongest benchmarks but 15.4 tok/s and 57s TTFT cold-start make it impractical for interactive agent work.
|
||||
|
||||
**GLM-5.2 status:** Not released. No official announcement from Z.AI as of June 2026.
|
||||
|
||||
---
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Model | Speed (tok/s) | TTFT | SWE-Bench V | SWE-Bench Pro | Terminal-Bench | MCP-Atlas | HLE | Code Arena Elo | Tool Reliability | Czech / Multilingual | Context | Multimodal | License | Verbosity | Known Bugs | Pricing (OpenRouter proxy) | Long-Horizon | Self-Host |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **glm-5.1:cloud** | ~198 | Low | ~58.4 (Pro) | 58.4 | 63.5 | 71.8 | 52.3 | 1530 | Excellent | No Czech claim; no drift observed | 200K (198K Ollama) | No | MIT | Low | None known | ~$4/M out | Proven | Yes |
|
||||
| **deepseek-v4-pro:cloud** | ~15.4 | 57s cold | 80.6 | — | 67.9 | 74.2 | 56.2 | — | Good | Strong multilingual (MMMLU 90.3) | 1M | No | MIT | Medium | Extreme variance | $1.74/M in | Unknown | Yes |
|
||||
| **deepseek-v4-flash:cloud** | ~30-50* | Moderate | ~75* | — | ~60* | — | — | — | Good | Strong multilingual | 1M | No | MIT | Medium | None known | $0.14/M in | Unknown | Yes |
|
||||
| **qwen3.5:397b-cloud** | ~10-20* | High | ~66-70 | — | ~59.3 | — | — | — | Good | **201 languages incl. Czech** | 1M | Yes | Apache 2.0 | Medium | "Too slow, accuracy issues" per user benchmark | — | Unknown | Yes |
|
||||
| **qwen3.5:cloud** | ~20-40* | Moderate | ~60-65 | — | ~55 | — | — | — | Good | 201 languages | 256K | Yes | Apache 2.0 | Medium | None known | — | Unknown | Yes |
|
||||
| **minimax-m3:cloud** | ~40-60* | Low | — | — | — | — | — | — | **Broken** | Undeclared | 512K | Yes | Open weights (pending) | — | **Tool result messages fail (#16389)** | $0.60/M in | N/A | Yes (pending) |
|
||||
| **kimi-k2.6:cloud** | ~30-50* | Moderate | 80.2 | — | 66.7 | — | 54.0 | — | Good | No Czech claim; **random Chinese drift** | 256K | Yes | Modified MIT | Medium | Chinese output bug; OR context bug (32K) | $0.60/M in | 200-300 tool calls | Yes |
|
||||
| **kimi-k2-thinking:cloud** | ~25-40* | Moderate | — | — | — | — | — | — | Good | No Czech claim | 256K | No | Modified MIT | High | Older (Nov 2025) | — | 200-300 seq tool calls | Yes |
|
||||
| **kimi-k2.5:cloud** | ~30-50* | Moderate | — | — | — | — | — | — | Good | No Czech claim | 256K | Yes | Modified MIT | Medium | Older (Jan 2026) | — | Unknown | Yes |
|
||||
| **nemotron-3-ultra:cloud** | ~50-80* | Low | ~60-79* | — | — | 74.2 | — | — | Unknown | Undeclared | 200K | No | NVIDIA Open | Low | Too new (Jun 4 2026) | $0.60/M in | Unknown | Yes (NVFP4) |
|
||||
| **nemotron-3-super:cloud** | ~60-100* | Low | 60.47 | — | — | — | — | — | Unknown | Undeclared | 1M | No | NVIDIA Open | Low | None known | — | Unknown | Yes |
|
||||
| **gemma4:31b-cloud** | ~80-120* | Low | ~52.0 | — | ~29.2 | — | — | — | Native FC | 140+ languages | 256K | Yes | Apache 2.0 | Low | None known | — | Unknown | Yes |
|
||||
| **devstral-2:123b-cloud** | ~40-60* | Moderate | 72.2 | — | 77.3 | — | — | — | Good | Undeclared | 128K | No | Apache 2.0 | Medium | None known | — | Unknown | Yes |
|
||||
| **gpt-oss:120b-cloud** | ~30-50* | Moderate | ~41.9 | — | — | — | — | — | Good | Undeclared | 128K | No | Apache 2.0 | Medium | Older (Aug 2025) | $0.039/M in | Unknown | Yes |
|
||||
| **gemini-3-flash-preview:cloud** | ~60-80* | Low | — | — | — | — | — | — | Good | Strong (Google) | 1M | Yes | Proprietary | Low | Older (Dec 2025) | — | Unknown | No |
|
||||
| **qwen3-coder-next:cloud** | ~40-60* | Moderate | ~70.6 | — | — | — | — | — | Good | 201 languages | 512K | No | Apache 2.0 | Medium | None known | — | Unknown | Yes |
|
||||
|
||||
*Speed estimates marked with * are inferred from similar-size MoE models or provider benchmarks, not direct Ollama Cloud measurements. GLM-5.1's ~198 tok/s is the only Ollama Cloud-specific speed figure found in the knowledge base.
|
||||
|
||||
---
|
||||
|
||||
## Detailed Analysis by Criterion
|
||||
|
||||
### 1. Speed (TTFT, throughput, wall-clock latency)
|
||||
- **GLM-5.1** is the clear speed leader on Ollama Cloud at ~198 tok/s.
|
||||
- **DeepSeek V4-Pro** is the slowest at ~15.4 tok/s with extreme variance and 57s cold-start TTFT.
|
||||
- **Gemma 4 31B** and **Nemotron 3 Super** are likely the fastest among alternatives due to small active parameter counts (31B dense, 12B active MoE).
|
||||
- **Qwen3.5:397B** is reported as "too slow" in user benchmarks.
|
||||
|
||||
### 2. Intelligence (Benchmarks)
|
||||
- **SWE-Bench Verified leaders:** DeepSeek V4-Pro (80.6%), Kimi K2.6 (80.2%), Devstral 2 (72.2%), Qwen3.5-397B (~66-70%), Nemotron 3 Super (60.47%), Gemma 4 31B (~52%), GPT-OSS 120B (~41.9%).
|
||||
- **SWE-Bench Pro:** GLM-5.1 leads open models at 58.4%.
|
||||
- **Terminal-Bench 2.0:** DeepSeek V4-Pro (67.9%), Kimi K2.6 (66.7%), Devstral 2 (77.3% — highest reported), GLM-5.1 (63.5%).
|
||||
- **MCP-Atlas:** Nemotron 3 Ultra (74.2%), DeepSeek V4-Pro (74.2%), GLM-5.1 (71.8%).
|
||||
- **HLE:** Kimi K2.6 (54.0%), GLM-5.1 (52.3%).
|
||||
- **Code Arena Elo:** GLM-5.1 at 1530 (#3 globally for agentic web dev).
|
||||
|
||||
### 3. Tool Calling Reliability & Schema Adherence
|
||||
- **GLM-5.1:** 99.6% schema adherence (per prior research), no known tool-calling failures.
|
||||
- **MiniMax M3:** **Critical blocker** — fails on tool result messages via Ollama Cloud OpenAI-compatible endpoint (issue #16389, 6 days old as of Jun 7). Returns empty responses.
|
||||
- **Kimi K2.6:** Good tool-call reliability but known OpenRouter context-length bug (reports 32K instead of 256K) that may affect Ollama.
|
||||
- **Gemma 4 31B:** Native function calling support.
|
||||
- **Nemotron 3 Ultra:** Too new; no verified tool-calling data yet.
|
||||
|
||||
### 4. Czech / Multilingual Support & Language Drift Risk
|
||||
- **Qwen3.5** (all variants): Explicitly claims 201 languages including Czech. Best documented multilingual support.
|
||||
- **Gemma 4 31B:** Claims 140+ languages, Apache 2.0.
|
||||
- **DeepSeek V4:** Strong multilingual (MMMLU 90.3, C-Eval 93.1) but no explicit Czech claim.
|
||||
- **GLM-5.1:** No explicit Czech claim, but **no known language drift** in practice.
|
||||
- **Kimi K2.6:** **Critical risk** — multiple user reports of random Chinese output even with English prompts. No explicit Czech support claim.
|
||||
- **MiniMax M3:** No explicit multilingual claim.
|
||||
- **Nemotron / Devstral / GPT-OSS:** No explicit Czech claims.
|
||||
|
||||
### 5. Context Window Size
|
||||
- **1M tokens:** deepseek-v4-pro, deepseek-v4-flash, nemotron-3-super, qwen3.5:397b-cloud, gemini-3-flash-preview
|
||||
- **512K:** minimax-m3, qwen3-coder-next
|
||||
- **256K:** kimi-k2.6, kimi-k2-thinking, kimi-k2.5, qwen3.5:cloud, gemma4:31b-cloud, devstral-2
|
||||
- **200K:** glm-5.1, nemotron-3-ultra
|
||||
- **128K:** gpt-oss:120b-cloud
|
||||
|
||||
### 6. Multimodality
|
||||
- **Multimodal:** minimax-m3, kimi-k2.6, kimi-k2.5, qwen3.5:397b-cloud, qwen3.5:cloud, gemma4:31b-cloud, gemini-3-flash-preview
|
||||
- **Text-only:** glm-5.1, deepseek-v4-pro, deepseek-v4-flash, kimi-k2-thinking, qwen3-coder-next, devstral-2, gpt-oss, nemotron-3-super/ultra
|
||||
|
||||
### 7. Open Weights & License
|
||||
- **MIT:** GLM-5.1, GLM-5, DeepSeek V4-Pro/Flash
|
||||
- **Apache 2.0:** Qwen3.5/Qwen3.6/Qwen3-coder, Gemma 4, GPT-OSS 120B, Devstral 2
|
||||
- **Modified MIT:** Kimi K2.6, Kimi K2-thinking, Kimi K2.5
|
||||
- **NVIDIA Open License:** Nemotron 3 Ultra/Super
|
||||
- **Proprietary:** Gemini-3-flash-preview
|
||||
- **MiniMax M3:** Open weights promised ~10 days after launch (early June 2026) — likely available by now.
|
||||
|
||||
### 8. Verbosity (Tokens per Answer)
|
||||
- **Low:** GLM-5.1 ("nejmenší verbosity z MoE rodiny"), Nemotron 3 Ultra/Super (up to 30% fewer tokens per turn), Gemma 4 31B
|
||||
- **Medium:** DeepSeek V4, Qwen3.5, Devstral 2, Kimi K2.6
|
||||
- **High:** Kimi K2-thinking (reasoning model)
|
||||
|
||||
### 9. Known Bugs / Blockers on Ollama Cloud
|
||||
- **MiniMax M3:** Tool result message failures (#16389) — **deploy blocker**.
|
||||
- **Kimi K2.6:** Random Chinese output drift — **deploy blocker for Czech use**.
|
||||
- **DeepSeek V4-Pro:** Extreme latency variance, 57s cold-start TTFT — usability issue.
|
||||
- **Qwen3.5:397B:** User-reported "too slow, accuracy issues" on Ollama Cloud.
|
||||
- **GLM-5.1:** No known bugs.
|
||||
|
||||
### 10. Pricing (OpenRouter proxy — Ollama Cloud is flat-rate $20/mo Pro)
|
||||
- **Cheapest input:** GPT-OSS 120B ($0.039/M), DeepSeek V4-Flash ($0.14/M)
|
||||
- **Mid-range:** DeepSeek V4-Pro ($1.74/M), MiniMax M3 ($0.60/M), Kimi K2.6 ($0.60/M), Nemotron 3 Ultra ($0.60/M)
|
||||
- **Most expensive:** GLM-5.1 (~$4/M output)
|
||||
- **Note:** Ollama Cloud Pro is flat-rate $20/month; per-token pricing only matters if switching to API/OpenRouter fallback.
|
||||
|
||||
### 11. Long-Horizon Agent Stability (Multi-turn, hundreds of tool calls)
|
||||
- **GLM-5.1:** Proven over "hundreds of rounds" — best sustained productivity per user experience.
|
||||
- **Kimi K2-thinking:** Explicitly designed for 200-300 sequential tool calls.
|
||||
- **Kimi K2.6:** Supports 200-300 sequential tool calls.
|
||||
- **Nemotron 3 Ultra:** Marketed for "long-running agents" but too new for verification.
|
||||
- **DeepSeek V4:** Unknown for sustained multi-turn agent use on Ollama Cloud.
|
||||
|
||||
### 12. Self-Host Fallback Possibility
|
||||
- **All models except Gemini-3-flash-preview** have open weights available on Hugging Face.
|
||||
- **NVFP4 quantization:** Nemotron 3 Ultra/Super require NVIDIA-specific formats.
|
||||
- **Hardware requirements:**
|
||||
- GLM-5.1: ~198K context on Ollama; self-host requires significant VRAM.
|
||||
- DeepSeek V4-Flash: 284B total / 13B active — efficient MoE, viable on consumer hardware.
|
||||
- Qwen3.5:397B: 397B total / 17B active — large but efficient.
|
||||
- Gemma 4 31B: Dense 31B — fits on 24GB GPU.
|
||||
- Devstral 2 123B: Large but coding-optimized.
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Primary Default (No Change)
|
||||
**`glm-5.1:cloud`** remains the best default nanobot agent model on Ollama Cloud.
|
||||
|
||||
**Why:**
|
||||
- Fastest measured speed (~198 tok/s)
|
||||
- Best proven track record for sustained agent sessions
|
||||
- MIT license
|
||||
- No known bugs or language drift
|
||||
- Strong benchmark suite (SWE-Bench Pro 58.4, Terminal-Bench 63.5, MCP-Atlas 71.8, Code Arena Elo 1530)
|
||||
- Low verbosity = lower token burn
|
||||
|
||||
### Alternative Tier 1 (Specific Needs)
|
||||
1. **`deepseek-v4-flash:cloud`** — Choose if you need 1M context for large codebase analysis or long-document processing. MIT license, open weights, cheaper than Pro. Tradeoff: slower than GLM-5.1 (~30-50 tok/s estimated).
|
||||
2. **`qwen3.5:397b-cloud`** — Choose if you need multimodal input (screenshots, diagrams) or explicit Czech language support (201 languages claimed). Apache 2.0, 1M context. Tradeoff: slow, reported accuracy issues on Ollama Cloud.
|
||||
|
||||
### Alternative Tier 2 (Niche Use)
|
||||
3. **`devstral-2:123b-cloud`** — Choose for pure coding-heavy workloads with strong benchmark scores (SWE-Bench 72.2%, Terminal-Bench 77.3%). Tradeoff: 128K context limit, no multimodal.
|
||||
4. **`gemma4:31b-cloud`** — Choose if you need a fast, lightweight alternative with native function calling and 140+ language support. Tradeoff: weaker agent benchmarks (SWE-Bench ~52%, Terminal-Bench ~29%).
|
||||
|
||||
### Avoid (Blockers)
|
||||
- **`minimax-m3:cloud`** — Tool-calling bug makes it unusable for agent work until Ollama fixes #16389.
|
||||
- **`kimi-k2.6:cloud`** — Chinese language drift is unacceptable for Czech-language agent use.
|
||||
- **`deepseek-v4-pro:cloud`** — 15.4 tok/s and 57s TTFT make it impractical for interactive agent sessions despite top benchmarks.
|
||||
|
||||
### Watch List
|
||||
- **`nemotron-3-ultra:cloud`** — Too new (released June 4, 2026). Promising specs (550B/55B, 1M ctx, low verbosity) but needs real-world agent validation on Ollama Cloud.
|
||||
- **`qwen3-coder-next:cloud`** — Strong coding focus, 512K context, Apache 2.0. Good candidate if coding is the primary workload.
|
||||
|
||||
---
|
||||
|
||||
## GLM-5.2 Status
|
||||
|
||||
**Not released.** As of June 7, 2026, Z.AI has made no official announcement of GLM-5.2. Reddit speculation from April 2026 suggested 50-83 days from GLM-5.1 launch (April 7, 2026), implying a June-July 2026 window, but no confirmation exists. It is not available on Ollama Cloud.
|
||||
|
||||
---
|
||||
|
||||
## Sources & Methodology
|
||||
|
||||
- Ollama Cloud model listings: ollama.com/search?c=cloud
|
||||
- Benchmark aggregators: llm-stats.com, benchlm.ai, benchmark.space, swebench.com
|
||||
- Vendor technical reports: NVIDIA Nemotron 3 Ultra (Jun 4, 2026), DeepSeek V4 (Apr 24, 2026), Qwen3.5/3.6 blog posts, Kimi K2.6 blog, Z.AI GLM-5.1 page
|
||||
- Community benchmarks: ollama-cloud-benchmark GitHub (erikwangz), dev.to user benchmarks
|
||||
- Bug trackers: ollama/ollama #16389 (MiniMax M3), Cursor/Reddit user reports (Kimi K2.6 Chinese drift)
|
||||
- Pricing: OpenRouter proxy rates (Ollama Cloud itself is flat-rate $20/mo Pro)
|
||||
|
||||
*Note on speed: Only GLM-5.1 has a direct Ollama Cloud speed measurement in our knowledge base (~198 tok/s). All other speed figures are estimates inferred from MoE active-parameter counts, provider benchmarks, or similar-platform measurements. Actual Ollama Cloud performance may vary due to load, cold starts, and quantization.*
|
||||
Reference in New Issue
Block a user