Chat Completions

Endpoint

POST /v1/chat/completions

Request Body

Field	Type	Default	Description
model	string	"auto"	Model identifier. Use "auto" for health-aware round-robin across all providers, or specify a model from GET /v1/models.
messages	array	required	Conversation history. Each item has role, content, and optional name.
stream	boolean	false	When true, responses are streamed as server-sent events.
temperature	number	—	Sampling temperature (0–2). Higher values produce more varied output.
max_tokens	number	—	Maximum number of tokens to generate.
min_reasoning_level	string	—	Minimum reasoning tier for auto-routing. One of "low", "medium", "high".
tools	array	—	List of tool/function definitions. When present, the gateway only routes to models that support tool calling.
tool_choice	string | object	—	Controls tool use: "none", "auto", "required", or { type: "function", function: { name: "..." } }.
response_format	object	—	{ type: "json_object" } requests JSON mode. { type: "json_schema", json_schema: { name, strict: true, schema } } requests schema output from qualified models.
project_id	string	required*	Project tag for analytics and rate accounting. You can also send x-gateway-project-id; one of the two is required.

Message Object

Messages support both text and multimodal (vision) content:

{
  "role": "user",
  "content": "What is the capital of France?",
  "name": "alice"
}

For vision requests, use the array content format with image URLs:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/photo.jpg" } }
  ]
}

When the gateway detects image_url content parts, it automatically filters to vision-capable models (e.g. Gemini, Groq Llama 4).

Field	Type	Description
role	string	One of "system", "user", "assistant", "tool".
content	string | array	Message text, or array of content parts for multimodal input.
name	string	Optional display name for the message author.

Capability-Based Routing

The gateway automatically detects what capabilities your request needs and filters models accordingly:

Request Feature	Detected Capability	Effect
tools array present	Tool calling	Only routes to models that support function calling
response_format: { type: "json_object" }	JSON mode	Only routes to models with structured output
image_url in message content	Vision	Only routes to vision-capable models
Large prompt	Context window	Excludes models whose context window is too small

This is fully automatic — no extra headers or configuration needed.

Non-Streaming Response

A standard OpenAI-compatible response object is returned. The gateway adds extra diagnostic headers:

Header	Description
x-gateway-provider	The backend provider that served the request (e.g. groq, gemini).
x-gateway-model	The exact model used by the provider.
x-gateway-attempts	Number of provider attempts before a successful response.
x-gateway-request-id	Unique request identifier for support and log correlation.
x-gateway-reasoning-effort	The reasoning effort level applied to the request.

Example Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1714000000,
  "model": "llama-3.3-70b-versatile",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 9,
    "total_tokens": 24
  }
}

Streaming Response

When stream: true, the response is a stream of data: lines in SSE format, each containing a JSON delta object. The stream is terminated by data: [DONE].

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1714000000,"model":"llama-3.3-70b-versatile","choices":[{"index":0,"delta":{"role":"assistant","content":"The"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1714000000,"model":"llama-3.3-70b-versatile","choices":[{"index":0,"delta":{"content":" capital"},"finish_reason":null}]}

data: [DONE]

Examples

Non-Streaming

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "What is the capital of France?" }
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

JavaScript

const response = await fetch('https://your-gateway.workers.dev/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer <GATEWAY_API_KEY>',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'auto',
    project_id: 'my_project',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: 'What is the capital of France?' },
    ],
    temperature: 0.7,
    max_tokens: 256,
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);
// Inspect gateway headers
console.log('Provider:', response.headers.get('x-gateway-provider'));
console.log('Model:', response.headers.get('x-gateway-model'));

Streaming

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [{ "role": "user", "content": "Tell me a short story." }],
    "stream": true
  }'

JavaScript

const response = await fetch('https://your-gateway.workers.dev/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer <GATEWAY_API_KEY>',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'auto',
    project_id: 'my_project',
    messages: [{ role: 'user', content: 'Tell me a short story.' }],
    stream: true,
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  for (const line of chunk.split('\n')) {
    if (!line.startsWith('data: ')) continue;
    const payload = line.slice(6).trim();
    if (payload === '[DONE]') break;

    const delta = JSON.parse(payload);
    const text = delta.choices?.[0]?.delta?.content ?? '';
    process.stdout.write(text);
  }
}

With Reasoning Effort

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [{ "role": "user", "content": "Solve: x^2 - 5x + 6 = 0" }],
    "min_reasoning_level": "high"
  }'

JavaScript

const response = await fetch('https://your-gateway.workers.dev/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer <GATEWAY_API_KEY>',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'auto',
    project_id: 'my_project',
    messages: [{ role: 'user', content: 'Solve: x^2 - 5x + 6 = 0' }],
    min_reasoning_level: 'high',
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);

Tool Calling (Agentic)

When you include tools, the gateway automatically routes to a model that supports function calling (Groq, Gemini, SambaNova, NVIDIA, Cerebras, or OpenRouter models with tool support).

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [{ "role": "user", "content": "What is the weather in San Francisco?" }],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": { "location": { "type": "string" } },
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

OpenAI SDK

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://your-gateway.workers.dev/v1',
  apiKey: '<GATEWAY_API_KEY>',
});

const completion = await client.chat.completions.create({
  model: 'auto',
  project_id: 'my_project',
  messages: [{ role: 'user', content: 'What is the weather in San Francisco?' }],
  tools: [{
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get current weather for a location',
      parameters: {
        type: 'object',
        properties: { location: { type: 'string' } },
        required: ['location'],
      },
    },
  }],
  tool_choice: 'auto',
});

const toolCall = completion.choices[0].message.tool_calls?.[0];
console.log(toolCall?.function.name, toolCall?.function.arguments);

JSON Mode (Structured Output)

When you set response_format, the gateway only picks models that support JSON mode.

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [{ "role": "user", "content": "List 3 programming languages with their year of creation as JSON" }],
    "response_format": { "type": "json_object" }
  }'

OpenAI SDK

const completion = await client.chat.completions.create({
  model: 'auto',
  project_id: 'my_project',
  messages: [{ role: 'user', content: 'List 3 programming languages with their year of creation as JSON' }],
  response_format: { type: 'json_object' },
});

const parsed = JSON.parse(completion.choices[0].message.content);
console.log(parsed);

Vision (Image Input)

Send images using the multimodal content format. The gateway auto-detects images and routes to vision-capable models (Gemini, Groq Llama 4).

curl

curl https://your-gateway.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer <GATEWAY_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "project_id": "my_project",
    "messages": [{
      "role": "user",
      "content": [
        { "type": "text", "text": "What is in this image?" },
        { "type": "image_url", "image_url": { "url": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg" } }
      ]
    }]
  }'

OpenAI SDK

const completion = await client.chat.completions.create({
  model: 'auto',
  project_id: 'my_project',
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Describe this image in detail.' },
      { type: 'image_url', image_url: { url: 'https://example.com/photo.jpg' } },
    ],
  }],
});

console.log(completion.choices[0].message.content);

Schema-constrained output

For non-streaming chat without tools, use response_format.type: "json_schema" with json_schema: { name: "classifier", strict: true, schema: { ... } }. The root schema must be an object with properties, required, and additionalProperties: false. Nested objects should also require every field and forbid additional properties. Keep schemas within the providers’ supported JSON Schema subset. Validate semantic values in the consumer as well.

Automatic routing retains health and cost ranking among qualified schema models: Gemini chat models and Groq GPT-OSS 20B/120B or Qwen 3.8 27B. Other models’ JSON-mode support alone does not qualify them. Token caps and the two-attempt gateway limit are unchanged. Generated json_validate_failed errors can use normal fallback; other upstream 400 caller errors remain non-retriable.