janhq/jan · error · Error

Tokenize request failed with status ${res.status}

Error message

Tokenize request failed with status ${res.status}

What it means

Thrown by countEmbeddingTokens() when the embedding model session's /tokenize HTTP endpoint returns a non-OK status. The session (sInfo) was resolved by ensureEmbeddingModelLoaded(), so a model is nominally loaded; the failure is that the in-process llamacpp server is not responding correctly to this specific request.

Source

Thrown at extensions/llamacpp-extension/src/index.ts:4174

   * on its session port. Char-based chunking can't reliably predict token
   * count (subword tokenizers vary widely by content), so callers that need
   * a hard guarantee against exceed_context_size_error should verify with
   * this rather than estimating from character length.
   */
  async countEmbeddingTokens(texts: string[]): Promise<number[]> {
    const sInfo = await this.ensureEmbeddingModelLoaded()
    const counts: number[] = []
    for (const text of texts) {
      const res = await fetch(`http://localhost:${sInfo.port}/tokenize`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${sInfo.api_key}`,
        },
        body: JSON.stringify({ content: text, model: sInfo.model_id }),
      })
      if (!res.ok) {
        throw new Error(`Tokenize request failed with status ${res.status}`)
      }
      const json = (await res.json()) as { tokens?: unknown[] }
      counts.push(Array.isArray(json.tokens) ? json.tokens.length : 0)
    }
    return counts
  }

  async embed(text: string[]): Promise<EmbeddingResponse> {
    const sInfo = await this.ensureEmbeddingModelLoaded()

    const ubatchSize =
      (this.config?.ubatch_size && this.config.ubatch_size > 0
        ? this.config.ubatch_size
        : 512) || 512
    const batches = buildEmbedBatches(text, ubatchSize)

    const attemptRequest = async (
      session: SessionInfo,

View on GitHub (pinned to fad3f12a14)

Solutions

1. Retry after a short delay or poll http://localhost:<port>/health until 200 before tokenizing.
2. Confirm the embedding model session is still loaded (findSessionByModel) and reload if it was unloaded.
3. Check that sInfo.api_key and sInfo.model_id match the running session.
4. Inspect the response body/status (401 vs 500) — 401 means auth, 5xx means the server process is unhealthy.

Example fix

// before
const res = await fetch(url, { method: 'POST', headers, body })
if (!res.ok) throw new Error(`Tokenize request failed with status ${res.status}`)

// after
const res = await fetch(url, { method: 'POST', headers, body })
if (!res.ok) {
  const detail = await res.text().catch(() => '')
  throw new Error(`Tokenize failed (${res.status}) on port ${sInfo.port}: ${detail}`)
}

async function tokenizeEndpointReady(port: number): Promise<boolean> {
  try {
    const r = await fetch(`http://localhost:${port}/health`)
    return r.ok
  } catch { return false }
}

if (!(await tokenizeEndpointReady(sInfo.port))) {
  throw new Error('Embedding /tokenize not ready; wait for health')
}

function hasSessionPort(s: unknown): s is { port: number; api_key: string; model_id: string } {
  return typeof s === 'object' && s !== null
    && typeof (s as any).port === 'number'
    && typeof (s as any).api_key === 'string'
    && typeof (s as any).model_id === 'string'
}

for (let attempt = 0; attempt < 3; attempt++) {
  const res = await fetch(url, { method: 'POST', headers, body })
  if (res.ok) { /* proceed */ break }
  if (res.status >= 500 && attempt < 2) { await delay(500 * (attempt + 1)); continue }
  throw new Error(`Tokenize failed ${res.status}`)
}

When it happens

Trigger: The embedding model process is starting up and not yet ready; the session crashed but findSessionByModel still returned stale info; an empty/oversized 'content' body the tokenizer rejects; the api_key header is wrong and returns 401/403.

Common situations: Calling countEmbeddingTokens immediately after load before /tokenize is live; concurrent unload invalidated the port; model_id in the body does not match the loaded session; port collision where another process now owns sInfo.port.

Related errors

• ubatch_size (${ubatchSize}) is too small. Minimum required:
• MLX API request failed with status ${response.status}: ${JSO
• Response body is null
• Checksum mismatch for ${name}; the download was corrupt or t
• Import of "${FALLBACK_EMBEDDING_MODEL_ID}" did not complete

AI-assisted analysis of janhq/jan@fad3f12a14 (2026-08-12). Data as JSON: /api/errors/cb97b956face6bac. Report an issue: GitHub.