⚡ August 23, 2026¶
Generated: 2026-08-23 03:01 UTC
Total Duration: 25m 58s
Iterations: 1
Judge (classifier) model: gpt-4.1
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 16 | 5 | 0 | 21 | 🟡 76% (16/21) |
| deepseek-v3.2-chat | 16 | 5 | 0 | 21 | 🟡 76% (16/21) |
| gemini-3.1-pro-preview | 17 | 4 | 0 | 21 | 🟡 81% (17/21) |
| gpt-5.3-codex | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| gpt-5.4 | 18 | 3 | 0 | 21 | 🟡 86% (18/21) |
| gpt-5.5 | 16 | 5 | 0 | 21 | 🟡 76% (16/21) |
| haiku-4.5 | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| opus-4.6 | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| opus-4.7 | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| opus-4.8 | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| qwen-next-80B-instruct | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| qwen-next-80B-thinking | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| sonnet-4.6 | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 18 | $0.01 | $0.00 | $0.04 | $0.25 |
| deepseek-v3.2-chat | 18 | $0.02 | $0.00 | $0.20 | $0.41 |
| gemini-3.1-pro-preview | 21 | $0.10 | $0.01 | $0.34 | $2.11 |
| gpt-5.4 | 21 | $0.06 | $0.01 | $0.11 | $1.21 |
| gpt-5.5 | 21 | $0.23 | $0.01 | $0.68 | $4.89 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 41.7 | 1.3 | 137.7 | 30.8 | 122.2 |
| deepseek-v3.2-chat | 43.3 | 1.2 | 308.0 | 19.6 | 146.0 |
| gemini-3.1-pro-preview | 30.7 | 6.6 | 92.9 | 25.2 | 92.9 |
| gpt-5.3-codex | 1.0 | 0.7 | 1.3 | 0.9 | 1.3 |
| gpt-5.4 | 15.1 | 3.9 | 32.2 | 14.2 | 25.4 |
| gpt-5.5 | 41.0 | 4.6 | 119.8 | 30.2 | 85.5 |
| haiku-4.5 | 1.3 | 0.8 | 6.0 | 1.1 | 1.4 |
| opus-4.6 | 1.6 | 0.9 | 6.1 | 1.3 | 1.8 |
| opus-4.7 | 1.6 | 0.9 | 6.0 | 1.3 | 2.0 |
| opus-4.8 | 1.4 | 1.0 | 1.8 | 1.3 | 1.6 |
| qwen-next-80B-instruct | 1.4 | 1.1 | 2.6 | 1.2 | 2.1 |
| qwen-next-80B-thinking | 1.3 | 0.9 | 2.3 | 1.3 | 1.6 |
| sonnet-4.6 | 1.4 | 0.9 | 6.0 | 1.2 | 2.0 |
⚠️ Note: 1 test(s) excluded from latency calculations due to throttling/timeout errors (gemini-3.1-pro-preview: 1)
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benchmark | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🔴 0% (0/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | |
| context_window | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| counting | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| datetime | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | |
| easy | 🟡 88% (⅞) | 🟡 88% (⅞) | 🟡 88% (⅞) | 🔴 0% (0/8) | 🟡 88% (⅞) | 🟡 62% (⅝) | 🔴 0% (0/8) | 🔴 0% (0/8) | 🔴 0% (0/8) | 🔴 0% (0/8) | 🔴 0% (0/8) | 🔴 0% (0/8) | 🔴 0% (0/8) | |
| elasticsearch | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 33% (⅓) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | |
| grafana | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| hard | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| kubernetes | 🟢 100% (10/10) | 🟢 100% (10/10) | 🟢 100% (10/10) | 🔴 0% (0/10) | 🟡 90% (9/10) | 🟡 90% (9/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | 🔴 0% (0/10) | |
| logs | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🔴 0% (0/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | 🔴 0% (0/6) | |
| loki | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| medium | 🟡 67% (6/9) | 🟡 67% (6/9) | 🟡 78% (7/9) | 🔴 0% (0/9) | 🟢 100% (9/9) | 🟡 89% (8/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | 🔴 0% (0/9) | |
| metrics | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| multi-cluster | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 33% (⅓) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | |
| network | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| port-forward | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | |
| question-answer | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| regression | 🟡 73% (11/15) | 🟡 73% (11/15) | 🟡 80% (12/15) | 🔴 0% (0/15) | 🟡 87% (13/15) | 🟡 80% (12/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | 🔴 0% (0/15) | |
| skills | 🟡 92% (11/12) | 🟡 92% (11/12) | 🟡 92% (11/12) | 🔴 0% (0/12) | 🟡 83% (10/12) | 🟡 67% (8/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | 🔴 0% (0/12) | |
| transparency | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 33% (⅓) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | 🔴 0% (0/3) | |
| Overall | 🟡 76% (16/21) | 🟡 76% (16/21) | 🟡 81% (17/21) | 🔴 0% (0/21) | 🟡 86% (18/21) | 🟡 76% (16/21) | 🔴 0% (0/21) | 🔴 0% (0/21) | 🔴 0% (0/21) | 🔴 0% (0/21) | 🔴 0% (0/21) | 🔴 0% (0/21) | 🔴 0% (0/21) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup failure (environment/infrastructure issue)
- ⏱️ Timeout or rate limit error
- ⏭️ Test skipped (e.g., known issue or precondition not met)
Detailed Raw Results¶
| Eval ID | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 41.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 19.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 23.8s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 12.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 21.9s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 2.1s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 2.0s |
| 100a_loki_historical_logs 🔗 | 🟢 100% (1/1) / ⏱️ 137.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 146.0s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 40.4s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 53.4s / 💰 $0.29 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s |
| 101_loki_historical_logs_pod_deleted 🔗 | 🟢 100% (1/1) / ⏱️ 122.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 36.3s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 25.2s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 1.3s | 🟢 100% (1/1) / ⏱️ 20.3s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 61.8s / 💰 $0.30 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.9s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 106.0s / 💰 $0.03 | 🔴 0% (0/1) / ⏱️ 103.2s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 58.4s / 💰 $0.20 | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 16.6s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 69.2s / 💰 $0.48 | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 0.9s |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 20.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 8.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 17.7s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 12.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 20.2s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.4s |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 36.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 18.7s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 15.0s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 19.9s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 0.9s |
| 176_network_policy_blocking_traffic_no_skills 🔗 | 🟢 100% (1/1) / ⏱️ 57.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 32.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 17.8s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 1.0s | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 85.5s / 💰 $0.64 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.0s |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (1/1) / ⏱️ 21.4s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 8.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 12.0s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 1.0s | 🟢 100% (1/1) / ⏱️ 6.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 9.0s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.7s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.0s |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 18.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 25.5s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 0.7s | 🔴 0% (0/1) / ⏱️ 7.0s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 23.2s / 💰 $0.13 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.7s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.0s |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 35.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 21.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 26.4s / 💰 $0.09 | 🔴 0% (0/1) / ⏱️ 1.3s | 🟢 100% (1/1) / ⏱️ 13.7s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 23.3s / 💰 $0.14 | 🔴 0% (0/1) / ⏱️ 6.0s | 🔴 0% (0/1) / ⏱️ 6.1s | 🔴 0% (0/1) / ⏱️ 6.0s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 2.6s | 🔴 0% (0/1) / ⏱️ 2.3s | 🔴 0% (0/1) / ⏱️ 6.0s |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 30.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 23.1s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 29.9s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 0.8s | 🟢 100% (1/1) / ⏱️ 14.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 30.2s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 2.0s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.5s |
| 254_elasticsearch_dr_test_log_check 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 524.0s / 💰 $0.14 | 🔴 0% (0/1) / ⏱️ 1.1s | 🟢 100% (1/1) / ⏱️ 23.1s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 119.8s / 💰 $0.68 | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.8s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.0s |
| 259_wrong_cluster_logs_confusion 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 92.9s / 💰 $0.31 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 18.2s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 55.2s / 💰 $0.29 | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.7s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.2s |
| 260_global_es_remote_cluster_logs 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.5s | 🟢 100% (1/1) / ⏱️ 28.2s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 32.2s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 69.0s / 💰 $0.41 | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.7s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (1/1) / ⏱️ 15.3s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 16.4s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 20.8s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 0.9s | 🟢 100% (1/1) / ⏱️ 11.7s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 14.8s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.2s |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 5.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 4.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 6.6s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 0.7s | 🟢 100% (1/1) / ⏱️ 3.9s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 4.6s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.8s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.1s |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 20.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 13.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.7s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 0.7s | 🟢 100% (1/1) / ⏱️ 7.9s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 20.1s / 💰 $0.09 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 0.9s |
| 61_exact_match_counting 🔗 | 🔴 0% (0/1) / ⏱️ 7.6s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 5.0s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 9.3s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 5.0s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 6.2s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.1s |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 56.3s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 308.0s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 23.9s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 0.8s | 🟢 100% (1/1) / ⏱️ 14.1s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 56.4s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 0.9s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.9s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.2s |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 74.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 58.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 37.1s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 1.3s | 🟢 100% (1/1) / ⏱️ 18.1s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 40.2s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 0.8s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.6s | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.4s | 🔴 0% (0/1) / ⏱️ 0.9s |
| 96_no_matching_skill 🔗 | 🟢 100% (1/1) / ⏱️ 65.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 63.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 84.0s / 💰 $0.34 | 🔴 0% (0/1) / ⏱️ 1.0s | 🟢 100% (1/1) / ⏱️ 22.1s / 💰 $0.09 | 🔴 0% (0/1) / ⏱️ 57.9s / 💰 $0.35 | 🔴 0% (0/1) / ⏱️ 1.0s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.8s | 🔴 0% (0/1) / ⏱️ 1.5s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.6s |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-32613070347.