{"record":{"id":"92720d3f75b0c126","repo":"hiyouga/LlamaFactory","slug":"failed-to-load-tokenizer","errorCode":null,"errorMessage":"Failed to load tokenizer.","messagePattern":"Failed to load tokenizer\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/model/loader.py","lineNumber":93,"sourceCode":"    \"\"\"\n    init_kwargs = _get_init_kwargs(model_args)\n    try:\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.model_name_or_path,\n            use_fast=model_args.use_fast_tokenizer,\n            split_special_tokens=model_args.split_special_tokens,\n            padding_side=\"right\",\n            **init_kwargs,\n        )\n    except ValueError:  # try another one\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.model_name_or_path,\n            use_fast=not model_args.use_fast_tokenizer,\n            padding_side=\"right\",\n            **init_kwargs,\n        )\n    except Exception as e:\n        raise OSError(\"Failed to load tokenizer.\") from e\n\n    patch_tokenizer(tokenizer, model_args)\n\n    try:\n        processor = AutoProcessor.from_pretrained(\n            model_args.model_name_or_path,\n            use_fast=model_args.use_fast_tokenizer,\n            **init_kwargs,\n        )\n    except ValueError:  # try another one\n        processor = AutoProcessor.from_pretrained(\n            model_args.model_name_or_path,\n            use_fast=not model_args.use_fast_tokenizer,\n            **init_kwargs,\n        )\n    except Exception as e:\n        logger.info_rank0(f\"Failed to load processor: {e}.\")\n        processor = None","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/loader.py#L75-L111","documentation":"Raised as OSError (wrapping the original exception) in _get_tokenizer when AutoTokenizer.from_pretrained fails with anything other than the specific ValueError that triggers the fast/slow retry. Typical root causes are a missing files.huggingface.co / network failure, an offline cache miss, a repo needing auth, or a path that has no tokenizer files; the original exception is chained via 'from e' for inspection.","triggerScenarios":"model_name_or_path points to a repo that 401/404s, HF_HUB_OFFLINE=1 without a cached tokenizer, a local dir lacking tokenizer_config.json, a revoked/gated model, or a transient network error during download.","commonSituations":"Corporate proxies blocking huggingface.co; expired or missing HF token for gated models (Llama etc.); typos in model names; partial local snapshots from interrupted downloads.","solutions":["Check the chained exception (raise ... from e) to see the real cause before retrying.","For auth issues: huggingface-cli login (or set HF_TOKEN) for gated repos.","For offline use: pre-download with huggingface-cli download <model> and/or set HF_HUB_OFFLINE=1 only after the cache is populated.","For local paths: verify tokenizer.json / tokenizer_config.json exist in the directory and the path is correct."],"exampleFix":"# before\nmodel_name_or_path: meta-llama/Llama-3-8B  # gated, not logged in\n\n# after\nhuggingface-cli login\nmodel_name_or_path: meta-llama/Llama-3-8B","handlingStrategy":"try-catch","validationCode":"import os\nfrom huggingface_hub import snapshot_download\n\npath = cfg[\"model_args\"][\"model_name_or_path\"]\nif not os.path.isdir(path):  # remote repo\n    snapshot_download(path, allow_patterns=[\"tokenizer*\", \"*.model\"])  # fails early with a clear hub error\nelse:\n    assert any(f.startswith(\"tokenizer\") or f == \"special_tokens_map.json\" for f in os.listdir(path)), \\\n        \"local dir has no tokenizer files\"","typeGuard":null,"tryCatchPattern":"from transformers import AutoTokenizer\ntry:\n    tok = AutoTokenizer.from_pretrained(model_path, padding_side=\"right\")\nexcept OSError as e:\n    cause = e.__cause__\n    if cause is not None and (\"401\" in str(cause) or \"gated\" in str(cause).lower()):\n        print(\"Auth problem: run huggingface-cli login\")\n    elif cause is not None and (\"offline\" in str(cause).lower() or \"Connection\" in str(cause)):\n        print(\"Network/offline problem: pre-download or unset HF_HUB_OFFLINE\")\n    raise","preventionTips":["Pre-download models with huggingface-cli download before training/serving runs.","Keep HF_TOKEN configured for gated repos and test it with a small request first.","Inspect the chained cause (__cause__) of the OSError; it distinguishes auth, network and missing-file failures."],"tags":["tokenizer","huggingface-hub","network","auth","offline"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}