{"record":{"id":"8a01e8f99343a8df","repo":"headroomlabs-ai/headroom","slug":"tiktoken-is-required-for-openai-provider-install","errorCode":null,"errorMessage":"tiktoken is required for OpenAI provider. Install with: pip install tiktoken","messagePattern":"tiktoken is required for OpenAI provider\\. Install with: pip install tiktoken","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"headroom/providers/openai.py","lineNumber":292,"sourceCode":"        return (\n            f\"OpenAI pricing data is {days_old} days old. \"\n            \"Cost estimates may be inaccurate. Verify against actual billing.\"\n        )\n    return None\n\n\n@lru_cache(maxsize=8)\ndef _get_encoding(encoding_name: str) -> Any:\n    \"\"\"Get tiktoken encoding, cached.\n\n    Routes through the bounded loader so a stalled vocab download raises\n    :class:`~headroom.tokenizers.tiktoken_counter.TiktokenLoadError` after a\n    timeout instead of hanging the caller indefinitely — ``tiktoken`` fetches\n    vocabularies with no network timeout, and this runs on whatever thread\n    first counts tokens for a model, including proxy startup (GH #956).\n    \"\"\"\n    if not TIKTOKEN_AVAILABLE:\n        raise RuntimeError(\n            \"tiktoken is required for OpenAI provider. Install with: pip install tiktoken\"\n        )\n    from ..tokenizers.tiktoken_counter import load_encoding\n\n    return load_encoding(encoding_name)\n\n\ndef _lookup_encoding_name(model: str, custom_encodings: dict[str, str] | None = None) -> str | None:\n    \"\"\"Resolve the tiktoken encoding for ``model``, or ``None`` if none claims it.\n\n    ``None`` is the \"not an OpenAI model\" signal: it means no explicit mapping,\n    no known prefix, and no OpenAI family pattern matched. Callers that can\n    reach a better tokenizer should use it rather than guess an encoding.\n    \"\"\"\n    # Check custom encodings first\n    if custom_encodings and model in custom_encodings:\n        return custom_encodings[model]\n","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/providers/openai.py#L274-L310","documentation":"The cached encoding loader in the OpenAI provider raises RuntimeError when tiktoken is not installed. Token counting for OpenAI models needs a tiktoken encoding; headroom marks tiktoken optional, and this guard fires on the first _get_encoding() call for any model. Note the loader is deliberately bounded and routes through load_encoding so a stalled vocab download raises TiktokenLoadError instead of hanging (GH #956) — but that only applies once tiktoken exists.","triggerScenarios":"Calling any token-counting path that resolves an encoding for an OpenAI-family model (e.g. the OpenAI provider's count_tokens / compression sizing) when TIKTOKEN_AVAILABLE is False, i.e. tiktoken was never installed in the runtime environment.","commonSituations":"Running the proxy with the OpenAI provider enabled but tiktoken omitted from requirements; slim Docker images that strip optional extras; deploying on a host where a different virtualenv is active than the one where tiktoken was installed.","solutions":["Install tiktoken in the environment that runs headroom: pip install tiktoken","Verify with: python -c \"import tiktoken\" to rule out a broken install or version conflict.","If you cannot install it, disable token-count-dependent features (compression/context sizing) for OpenAI models or use a provider whose tokenizer is available."],"exampleFix":"# before\nn = provider.count_tokens(\"gpt-4o\", text)  # RuntimeError: tiktoken is required...\n\n# after (shell)\n# pip install tiktoken\nn = provider.count_tokens(\"gpt-4o\", text)  # ok","handlingStrategy":"try-catch","validationCode":"import importlib.util\n\ntiktoken_ready = importlib.util.find_spec(\"tiktoken\") is not None","typeGuard":"def tiktoken_available() -> bool:\n    return importlib.util.find_spec(\"tiktoken\") is not None","tryCatchPattern":"try:\n    enc = _get_encoding(encoding_name)\nexcept RuntimeError as exc:\n    if \"pip install tiktoken\" in str(exc):\n        raise SystemExit(\"Install tiktoken to enable OpenAI token counting\") from exc\n    raise","preventionTips":["Include tiktoken in environments that enable OpenAI compression/context features.","Exercise a token-count call in CI for every provider you configure, so missing optional deps fail in CI not production.","Remember the separate failure mode: once tiktoken exists, vocab download issues raise TiktokenLoadError, not this RuntimeError."],"tags":["dependency","installation","tokenizer","openai","python"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}