dotnet/machinelearning · error · InvalidOperationException

The content of the vocabulary file '{vocabFile}' is not vali

Error message

The content of the vocabulary file '{vocabFile}' is not valid.

What it means

After successfully opening and deserializing the vocabulary file, BpeOptions throws InvalidOperationException if JsonSerializer.Deserialize<Dictionary<string,int>> returns null. This means the JSON stream deserialized to null (e.g. the literal 'null' or an empty document), so no valid vocabulary could be built. It indicates corrupt or wrong content in a file that does exist.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs:54

        /// <param name="mergesFile">The file path containing the tokens's pairs list.</param>
        public BpeOptions(string vocabFile, string? mergesFile = null)
        {
            if (vocabFile is null)
            {
                throw new ArgumentNullException(nameof(vocabFile));
            }

            if (!File.Exists(vocabFile))
            {
                throw new ArgumentException($"Could not find the vocabulary file '{vocabFile}'.");
            }

            using Stream vocabStream = File.OpenRead(vocabFile);
            Dictionary<string, int>? dictionary = JsonSerializer.Deserialize<Dictionary<string, int>>(vocabStream);

            if (dictionary is null)
            {
                throw new InvalidOperationException($"The content of the vocabulary file '{vocabFile}' is not valid.");
            }

            Vocabulary = dictionary;

            if (mergesFile is not null)
            {
                if (!File.Exists(mergesFile))
                {
                    throw new ArgumentException($"Could not find the merges file '{mergesFile}'.");
                }

                using Stream mergesStream = File.OpenRead(mergesFile);
                using StreamReader reader = new(mergesStream);

                List<string> merges = new();

                int lineNumber = 0;
                string? line;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Inspect vocab.json and ensure it contains a valid JSON object mapping strings to integer ids, e.g. {"<unk>": 0, "hello": 1}.
2. Re-download or restore the vocabulary file from the model's official repository.
3. Check the file is not empty or a Git LFS pointer; run the appropriate LFS pull if so.
4. Deserialize the file yourself with JsonSerializer first to validate the content before passing the path to BpeOptions.

Example fix

// before (vocab.json contains: null)
var options = new BpeOptions("vocab.json");
// after: validate content first
var vocab = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText("vocab.json"));
if (vocab is null || vocab.Count == 0) throw new InvalidDataException("vocab.json is not a valid vocab dictionary");
var options = new BpeOptions("vocab.json");

var check = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText(vocabPath));
if (check is null || check.Count == 0) throw new InvalidDataException($"{vocabPath} is not a valid vocab JSON object");

try { var options = new BpeOptions(vocabPath); } catch (InvalidOperationException ex) { logger.LogError(ex, "Corrupt vocab file {Path}", vocabPath); }

When it happens

Trigger: vocab.json exists but contains 'null', is empty, or the deserializer yields no dictionary; a placeholder/zero-byte file was committed or downloaded partially.

Common situations: Interrupted model download left an empty file; a file of a different format was renamed to vocab.json; a Git LFS pointer file was checked out without fetching the actual blob.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Problems met when parsing JSON vocabulary object.{Environmen
• Failed to read the vocabulary file.
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/60f02ea31803ad5f. Report an issue: GitHub.