{"record":{"id":"82d7c555460bfd26","repo":"janhq/jan","slug":"the-request-exceeds-the-available-context-size","errorCode":null,"errorMessage":"the request exceeds the available context size.","messagePattern":"the request exceeds the available context size\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"extensions/mlx-extension/src/index.ts","lineNumber":411,"sourceCode":"\n    const response = await fetch(url, {\n      method: 'POST',\n      headers,\n      body,\n      signal: abortController?.signal,\n    })\n\n    if (!response.ok) {\n      const errorData = await response.json().catch(() => null)\n      throw new Error(\n        `MLX API request failed with status ${response.status}: ${JSON.stringify(errorData)}`\n      )\n    }\n\n    const completionResponse = (await response.json()) as chatCompletion\n\n    if (completionResponse.choices?.[0]?.finish_reason === 'length') {\n      throw new Error(OUT_OF_CONTEXT_SIZE)\n    }\n\n    return completionResponse\n  }\n\n  private async *handleStreamingResponse(\n    url: string,\n    headers: HeadersInit,\n    body: string,\n    abortController?: AbortController\n  ): AsyncIterable<chatCompletionChunk> {\n    // AbortSignal.any() is not available in all runtimes (e.g. WebKit/JavaScriptCore),\n    // so we manually combine the timeout and external abort signals.\n    const combinedController = new AbortController()\n    const timeoutId = setTimeout(\n      () => combinedController.abort(new Error('Request timed out')),\n      this.timeout * 1000\n    )","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/janhq/jan/blob/fad3f12a147d138388a66f0d92a02b2675f65294/extensions/mlx-extension/src/index.ts#L393-L429","documentation":"Thrown by chat() (non-streaming) when the completion returns finish_reason === 'length'. That finish reason means generation stopped because it hit the output token cap (max_tokens) or the combined prompt+output reached the context window. The constant OUT_OF_CONTEXT_SIZE is the shared message text. This is a soft, content-level limit — the HTTP request itself succeeded (200).","triggerScenarios":"Prompt plus max_tokens exceeds n_ctx; a long system prompt + conversation leaves little room for output; max_tokens set too high relative to context; RAG injected too many retrieved chunks.","commonSituations":"Long conversations in a small ctx_size model; aggressive max_tokens; large tool descriptions eating context; the model legitimately wanted to generate more than the cap allowed.","solutions":["Increase ctx_size when loading the model, or reduce max_tokens in the request.","Trim conversation history / system prompt / retrieved context before sending.","Treat this as recoverable: the partial output is valid, surface it and offer to continue.","Switch to a model with a larger native context window."],"exampleFix":"// before\n// opts.max_tokens left at a large default with a small ctx_size model\n\n// after\nconst totalBudget = sessionInfo.nCtx ?? 4096\nopts.max_tokens = Math.min(opts.max_tokens ?? 512, totalBudget - estimatedPromptTokens - 64)","handlingStrategy":"validation","validationCode":"const nCtx = sessionInfo.nCtx ?? 4096\nconst estPrompt = estimateTokens(JSON.stringify(opts.messages))\nopts.max_tokens = Math.min(opts.max_tokens ?? 512, Math.max(1, nCtx - estPrompt - 64))","typeGuard":null,"tryCatchPattern":"try {\n  return await engine.chat(opts, abort)\n} catch (e) {\n  if (/exceeds the available context size/.test(String(e))) {\n    opts.max_tokens = Math.floor((opts.max_tokens ?? 512) / 2)\n    return await engine.chat(opts, abort)\n  }\n  throw e\n}","preventionTips":["Set max_tokens relative to remaining context (nCtx - prompt).","Trim conversation history and RAG context before sending.","Increase ctx_size at load for long conversations."],"tags":["context-limit","chat","finish-reason","mlx","typescript"],"backgroundTag":null,"analyzedSha":"fad3f12a147d138388a66f0d92a02b2675f65294","analyzedAt":"2026-08-12T20:33:47.516Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}