Mintplex-Labs/anything-llm · error · Error

Foundry Local crashed trying to reply to this message. You s

Error message

Foundry Local crashed trying to reply to this message. You should change the message or try again.

What it means

In `#handleFunctionCallChat` (the UnTooled non-streaming callback), when the OpenAI SDK throws and `#isPrematureClose(error)` is true (error.code === 'ERR_STREAM_PREMATURE_CLOSE' or message matches /premature close/i), the catch calls `FoundryLLM.explainStreamError(e, model)`. That static method deletes the model from the in-process loaded-models cache (Foundry Local answers 200 then drops the socket when its model has crashed/unloaded, so the next call must reload it) and returns the literal 'Foundry Local crashed trying to reply to this message. You should change the message or try again.' Any non-premature error in this path returns `null`, surfacing elsewhere as an empty result.

Source

Thrown at server/utils/agents/aibitat/providers/foundry.js:136

  async #handleFunctionCallChat({ messages = [] }) {
    await this.#assertContextLimits();
    return await this.client.chat.completions
      .create({
        model: this.model,
        messages,
        max_completion_tokens: FoundryLLM.promptWindowLimit(this.model),
      })
      .then((result) => {
        if (!result.hasOwnProperty("choices"))
          throw new Error("Microsoft Foundry Local chat: No results!");
        if (result.choices.length === 0)
          throw new Error("Microsoft Foundry Local chat: No results length!");
        return result.choices[0].message.content;
      })
      .catch((e) => {
        if (this.#isPrematureClose(e))
          throw new Error(FoundryLLM.explainStreamError(e, this.model));
        return null;
      });
  }

  async #handleFunctionCallStream({ messages = [] }) {
    await this.#assertContextLimits();
    return await this.client.chat.completions.create({
      model: this.model,
      stream: true,
      messages,
      max_completion_tokens: FoundryLLM.promptWindowLimit(this.model),
    });
  }

  /**
   * Stream a chat completion with tool calling support.
   * Uses native tool calling when supported, otherwise falls back to UnTooled.
   */

View on GitHub (pinned to 526360e320)

Solutions

1. Retry the turn — explainStreamError already evicted the model from cache so the next call reloads it.
2. Shorten or simplify the prompt to avoid local OOM.
3. Confirm the Foundry Local service is healthy (`foundry` CLI / Windows service status).
4. Reduce concurrency against the local Foundry instance to one in-flight request.
5. If it recurs, re-pull/reload the model with the Foundry CLI and check GPU/driver health.

// Preflight: confirm Foundry Local has the model loaded before the chat callback.
async function assertFoundryModelLoaded(model) {
  await new FoundryLLM(null, model).assertModelLoaded();
}

// Detect the premature-close shape that triggers this branch.
function isFoundryPrematureClose(e) {
  return e?.code === 'ERR_STREAM_PREMATURE_CLOSE' || /premature close/i.test(e?.message ?? '');
}

// The model is auto-evicted on premature close — retry once; it will reload.
try { return await provider.complete(messages, functions); }
catch (e) {
  if (isFoundryPrematureClose(e)) return await retryAfterShortBackoff();
  throw e;
}

When it happens

Trigger: Microsoft Foundry Local crashed the model process mid-reply; the model OOM'd on the prompt; the Foundry Local service restarted; a GPU/driver fault aborted local inference; the model was unloaded by a concurrent request.

Common situations: Large/long prompts causing local model OOM; Foundry Local beta instability; concurrent requests to a single-model local instance; GPU driver/firmware issue; Foundry Local upgraded and the model needs re-download.

Related errors

• error.message
• No Foundry Base Path was set.
• Could not load ${this.model} into Foundry Local: ${error}
• error.message
• res.statusText || "Error finding custom models."

AI-assisted analysis of Mintplex-Labs/anything-llm@526360e320 (2026-08-13). Data as JSON: /api/errors/dc93bb1b49459048. Report an issue: GitHub.