# Free AI - Benchmark & Cost Optimizer

> Fixture-backed model benchmark matrix, workload recommendations and public experiment fixtures. Browser-local snapshots and experiment mutations are excluded.

Canonical: https://ai-gateway.sassmaker.com/benchmark

Observed: 2026-10-27T20:00:46.819Z

Prototype operator surface. Benchmark rows are a local fixture. Compare routing experiments before changing production weights.

Fixture: benchmark-cost-optimizer-v1; source: fixture; generated: 2026-06-04T12:00:00.000Z.

Candidates: 5; workloads: 4; average success: 86.4%; direct API cost: $0.004.

## Model benchmark matrix

| Model | Provider | Cost /1M | p50 | p90 | Success | Quality | Cooldown | Status | Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| groq/llama-3.3-70b-versatile | groq | $0 | 420 ms | 780 ms | 94.0% | high | — | available | 0.91 |
| gemini/gemini-2.0-flash | gemini | $0 | 680 ms | 1200 ms | 97.0% | medium | — | available | 0.89 |
| workers_ai/@cf/meta/llama-3.1-8b-instruct | workers_ai | $0.020 | 1100 ms | 2100 ms | 88.0% | low | — | available | 0.76 |
| cerebras/llama3.1-8b | cerebras | $0 | 310 ms | 520 ms | 72.0% | low | — | cooldown | 0.58 |
| openrouter/meta-llama/llama-3.2-3b-instruct:free | openrouter | $0 | 890 ms | 1600 ms | 81.0% | low | — | degraded | 0.42 |


## Recommended route by workload

### Text chat

text

Short assistant turns, no tools or vision

Recommended: groq/llama-3.3-70b-versatile — Highest score at $0 direct cost with sub-500ms p50

- gemini/gemini-2.0-flash — Higher success rate, slightly slower
- workers_ai/@cf/meta/llama-3.1-8b-instruct — Neuron budget fallback when external quotas tighten

### Tool-calling agent

text+tools

Function-calling loops with structured outputs

Recommended: gemini/gemini-2.0-flash — Stable tool calling and JSON mode on free tier

- groq/llama-3.3-70b-versatile — Faster when tool schema stays simple

### Vision describe

text+vision

Image + text multimodal prompts

Recommended: gemini/gemini-2.0-flash — Vision-capable with best success rate in fixture

- workers_ai/@cf/meta/llama-3.1-8b-instruct — Not vision-capable — excluded by capability filter

### JSON extraction

json+text

response_format json_object workloads

Recommended: groq/llama-3.3-70b-versatile — json_mode capable with lowest latency among capable models

- gemini/gemini-2.0-flash — Slightly higher reliability for long JSON payloads

## Experiment ledger

Public fixture entries only. Browser-local snapshots are private and excluded.

| When | Label | Change | Success | Latency | Cost /1k | Fallback |
| --- | --- | --- | --- | --- | --- | --- |
| 2026-06-04T12:00:00.000Z | Latency bias trial | latency weight 0.15 → 0.22 (prototype only) | 87.0% | 610 ms | $0 | 14.0% |
| 2026-06-01T09:00:00.000Z | Baseline scoring weights | success_rate×0.6 + headroom×0.2 + latency×0.15 | 89.0% | 740 ms | $0 | 11.0% |


## Record experiment entry

The HTML form accepts a label, change, baseline and notes. Reading this page does not create experiments.


## Related
- [Gateway health](https://ai-gateway.sassmaker.com/health)
- [Model catalog](https://ai-gateway.sassmaker.com/models)
- [Traffic dashboard](https://ai-gateway.sassmaker.com/dashboard)
- [API docs](https://ai-gateway.sassmaker.com/docs/)