janhq/jan · warning · Error

the request exceeds the available context size.

Error message

the request exceeds the available context size.

What it means

Thrown by chat() (non-streaming) when the completion returns finish_reason === 'length'. That finish reason means generation stopped because it hit the output token cap (max_tokens) or the combined prompt+output reached the context window. The constant OUT_OF_CONTEXT_SIZE is the shared message text. This is a soft, content-level limit — the HTTP request itself succeeded (200).

Source

Thrown at extensions/mlx-extension/src/index.ts:411

    const response = await fetch(url, {
      method: 'POST',
      headers,
      body,
      signal: abortController?.signal,
    })

    if (!response.ok) {
      const errorData = await response.json().catch(() => null)
      throw new Error(
        `MLX API request failed with status ${response.status}: ${JSON.stringify(errorData)}`
      )
    }

    const completionResponse = (await response.json()) as chatCompletion

    if (completionResponse.choices?.[0]?.finish_reason === 'length') {
      throw new Error(OUT_OF_CONTEXT_SIZE)
    }

    return completionResponse
  }

  private async *handleStreamingResponse(
    url: string,
    headers: HeadersInit,
    body: string,
    abortController?: AbortController
  ): AsyncIterable<chatCompletionChunk> {
    // AbortSignal.any() is not available in all runtimes (e.g. WebKit/JavaScriptCore),
    // so we manually combine the timeout and external abort signals.
    const combinedController = new AbortController()
    const timeoutId = setTimeout(
      () => combinedController.abort(new Error('Request timed out')),
      this.timeout * 1000
    )

View on GitHub (pinned to fad3f12a14)

Solutions

Increase ctx_size when loading the model, or reduce max_tokens in the request.
Trim conversation history / system prompt / retrieved context before sending.
Treat this as recoverable: the partial output is valid, surface it and offer to continue.
Switch to a model with a larger native context window.

Example fix

// before
// opts.max_tokens left at a large default with a small ctx_size model

// after
const totalBudget = sessionInfo.nCtx ?? 4096
opts.max_tokens = Math.min(opts.max_tokens ?? 512, totalBudget - estimatedPromptTokens - 64)

Defensive patterns

Strategy: validation

Validate before calling

const nCtx = sessionInfo.nCtx ?? 4096
const estPrompt = estimateTokens(JSON.stringify(opts.messages))
opts.max_tokens = Math.min(opts.max_tokens ?? 512, Math.max(1, nCtx - estPrompt - 64))

Try / catch

try {
  return await engine.chat(opts, abort)
} catch (e) {
  if (/exceeds the available context size/.test(String(e))) {
    opts.max_tokens = Math.floor((opts.max_tokens ?? 512) / 2)
    return await engine.chat(opts, abort)
  }
  throw e
}

Prevention

Set max_tokens relative to remaining context (nCtx - prompt).
Trim conversation history and RAG context before sending.
Increase ctx_size at load for long conversations.

When it happens

Trigger: Prompt plus max_tokens exceeds n_ctx; a long system prompt + conversation leaves little room for output; max_tokens set too high relative to context; RAG injected too many retrieved chunks.

Common situations: Long conversations in a small ctx_size model; aggressive max_tokens; large tool descriptions eating context; the model legitimately wanted to generate more than the cap allowed.

Related errors

AI-assisted analysis of janhq/jan@fad3f12a14 (2026-08-12). Data as JSON: /api/errors/82d7c555460bfd26. Report an issue: GitHub.