What causes "No NVIDIA NIM token context limit was set." in Mintplex-Labs/anything-llm?

Set NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT to a plain integer (e.g. 32768) or remove it to use the 4096 default

Mintplex-Labs/anything-llm · error · Error

No NVIDIA NIM token context limit was set.

Error message

No NVIDIA NIM token context limit was set.

What it means

Static variant of NvidiaNimLLM.promptWindowLimit(): it reads NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT with a 4096 fallback and throws when the value cannot be converted to a number. Since unset/empty falls back to 4096, the throw means the variable is set to a non-numeric string. The static form can be invoked without an instance (e.g. for preflight sizing), before any model sync has normalized the value.

Solutions

Set NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT to a plain integer (e.g. 32768) or remove it to use the 4096 default
Let the provider's model sync populate it from the model's max_model_len by not overriding it manually
Restart the server after fixing .env
Validate numeric env vars at startup

Example fix

# before
NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT=32k
# after
NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT=32768

Defensive patterns

Strategy: validation

Validate before calling

const raw = process.env.NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT || '4096';
const limit = Number(raw);
if (!Number.isFinite(limit) || limit <= 0)
  throw new Error(`NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT must be an integer, got: ${raw}`);
const window = NvidiaNimLLM.promptWindowLimit(modelName);

Try / catch

try {
  const limit = NvidiaNimLLM.promptWindowLimit(modelName);
} catch (e) {
  if (e.message.includes('token context limit'))
    return respondConfigError(e); // fix the env value, do not retry
  throw e;
}

Prevention

Use bare integers for token-limit env vars
Let the provider auto-populate the limit from the model's max_model_len instead of setting it manually
Validate numeric env vars at process startup

When it happens

Trigger: Setting NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT to a non-integer like '8k' or '32K tokens'; calling NvidiaNimLLM.promptWindowLimit() (static) before the provider's model sync has overwritten the env var with a numeric max_model_len.

Common situations: Human-friendly values in .env; quotes/units copied from a model card; values with trailing spaces in compose files.

Understand the failure class

Background: "is not a valid" / "Invalid ... value" environment variable errors: how libraries validate env vars and what to do when they reject yours — this error's family across 48 libraries.

Related errors

AI-assisted analysis of Mintplex-Labs/anything-llm@3aec848f28 (2026-08-18). Data as JSON: /api/errors/a8ceeb62b9ee349e. Report an issue: GitHub.

Appendix: source

Thrown at server/utils/AiProviders/nvidiaNim/index.js:89

        return [];
      });

    if (!model.length) return;
    const modelInfo = model.find((model) => model.id === modelId);
    if (!modelInfo) return;
    process.env.NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT = Number(
      modelInfo.max_model_len || 4096
    );
  }

  streamingEnabled() {
    return "streamGetChatCompletion" in this;
  }

  static promptWindowLimit(_modelName) {
    const limit = process.env.NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT || 4096;
    if (!limit || isNaN(Number(limit)))
      throw new Error("No NVIDIA NIM token context limit was set.");
    return Number(limit);
  }

  // Ensure the user set a value for the token limit
  // and if undefined - assume 4096 window.
  promptWindowLimit() {
    const limit = process.env.NVIDIA_NIM_LLM_MODEL_TOKEN_LIMIT || 4096;
    if (!limit || isNaN(Number(limit)))
      throw new Error("No NVIDIA NIM token context limit was set.");
    return Number(limit);
  }

  async isValidChatCompletionModel(_ = "") {
    return true;
  }

  /**
   * Generates appropriate content array for a message + attachments.

View on GitHub (pinned to 3aec848f28)