dotnet/machinelearning · error · ArgumentException

Failed to read the vocabulary file.

Error message

Failed to read the vocabulary file.

What it means

Thrown when the vocabulary JSON deserializes successfully but yields a null result (JsonSerializer.Deserialize returns null, e.g. for a stream containing 'null' or an empty payload consumed as JSON null). The library requires a non-null vocabulary dictionary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:193

                highestOccurrenceMappingStream.Dispose();
            }
        }

        private static Dictionary<StringSpanOrdinalKey, int> GetVocabulary(Stream vocabularyStream)
        {
            Dictionary<StringSpanOrdinalKey, int>? vocab;
            try
            {
                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.DictionaryStringSpanOrdinalKeyInt32);
            }
            catch (Exception e)
            {
                throw new ArgumentException($"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}");
            }

            if (vocab is null)
            {
                throw new ArgumentException($"Failed to read the vocabulary file.");
            }

            return vocab;
        }

        private static Cache<(string, string), int> GetMergeRanks(Stream mergeStream)
        {
            var mergeRanks = new Cache<(string, string), int>(60_000);
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file
                if (reader.Peek() >= 0)
                {
                    string ignored = reader.ReadLine()!;
                }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Check the vocabulary stream length is non-zero and contains a JSON object before constructing the tokenizer.
2. Re-download or regenerate vocab.json and verify it starts with '{'.
3. Wrap the stream in a StreamReader and assert the first non-whitespace character is '{'.

Example fix

// before
var tokenizer = new EnglishRobertaTokenizer(maybeEmptyStream, mergesStream);
// after
if (maybeEmptyStream.Length == 0) throw new InvalidOperationException("vocabulary file is empty");
var tokenizer = new EnglishRobertaTokenizer(maybeEmptyStream, mergesStream);

if (vocabStream.Length == 0) throw new InvalidDataException("Vocabulary file is empty");
vocabStream.Seek(0, SeekOrigin.Begin);
int first = vocabStream.ReadByte();
vocabStream.Seek(0, SeekOrigin.Begin);
if (first != '{') throw new InvalidDataException("Vocabulary file is not a JSON object");

try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }
catch (ArgumentException ex) when (ex.Message.Contains("Failed to read the vocabulary file")) { throw new InvalidDataException("vocab.json deserialized to null; check file contents", ex); }

When it happens

Trigger: Passing an empty vocabularyStream or a stream whose entire content is the JSON literal 'null' to the EnglishRobertaTokenizer constructor.

Common situations: A zero-byte vocab.json produced by a failed download; a serialization pipeline that wrote 'null'; passing MemoryStream that was never written to.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• The content of the vocabulary file '{vocabFile}' is not vali
• Problems met when parsing JSON vocabulary object.{Environmen
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cbd1dc150102b129. Report an issue: GitHub.