dotnet/machinelearning · error · ArgumentException

Failed to read the vocabulary file.

Error message

Failed to read the vocabulary file.

What it means

If JsonSerializer.Deserialize returns null for the vocabulary stream, the loader throws ArgumentException('Failed to read the vocabulary file.'). This occurs when the JSON deserializes to null (e.g. the stream contains the literal 'null') or yields no usable vocabulary object.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1761

            { "\t\t\t",                             50293 },
            { "\t\t",                               50294 },
        };

        private static Dictionary<StringSpanOrdinalKey, (int, string)> GetVocabulary(Stream vocabularyStream)
        {
            Vocabulary? vocab;
            try
            {
                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.Vocabulary);
            }
            catch (Exception e)
            {
                throw new ArgumentException($"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}");
            }

            if (vocab is null)
            {
                throw new ArgumentException($"Failed to read the vocabulary file.");
            }

            return vocab;
        }

        internal static Dictionary<StringSpanOrdinalKeyPair, int> GetMergeRanks(Stream mergeStream)
        {
            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file
                if (reader.Peek() >= 0)
                {
                    string ignored = reader.ReadLine()!;
                }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Inspect the stream content: it must be a valid JSON token-to-id mapping, not 'null' or empty.
2. Verify the file is the model's vocab.json and re-download if it is empty or a placeholder.
3. Check stream position/length before passing (Length > 0, Position == 0).
4. Add a pre-flight JSON deserialization in your code to fail with a clearer message.

Example fix

// before
using var s = File.OpenRead("vocab.json"); // 0 bytes -> deserializes to null
// after
if (new FileInfo("vocab.json").Length == 0) throw new InvalidOperationException("vocab.json is empty");
using var s = File.OpenRead("vocab.json");

if (new FileInfo(vocabPath).Length == 0) throw new InvalidOperationException("vocab.json is empty");
// or after load: if (JsonSerializer.Deserialize<Vocabulary>(json) is null) throw ...

bool IsUsableVocab(Stream s) { try { return JsonSerializer.Deserialize<Dictionary<string,int>>(s) is { Count: > 0 }; } catch { return false; } }

try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }
catch (ArgumentException ex) when (ex.Message == "Failed to read the vocabulary file.") { /* re-fetch vocab.json */ }

When it happens

Trigger: Passing a vocabulary stream whose JSON payload deserializes to null — an empty-ish file containing 'null', or content that does not map onto the Vocabulary shape expected by ModelSourceGenerationContext.

Common situations: Placeholder files checked into source control ('null' or empty), zero-byte downloads, or wrong file supplied as the vocabulary.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• The beginning of sentence token '{BeginningOfSentenceToken}'
• The end of sentence token '{EndOfSentenceToken}' is not foun
• Problems met when parsing JSON vocabulary object.{Environmen
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/4caf65aba060e920. Report an issue: GitHub.