dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(vocabularyPath));

Error message

throw new ArgumentNullException(nameof(vocabularyPath));

What it means

This CodeGenTokenizer constructor overload takes vocabularyPath and mergePath file paths and throws ArgumentNullException from the constructor initializer when vocabularyPath is null. Unlike BpeOptions, there is no File.Exists check here; null is the only thing rejected before the file is opened. It fails fast before File.OpenRead would throw a different exception.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:67

        /// <param name="addPrefixSpace">Indicate whether to include a leading space before encoding the text.</param>
        /// <param name="addBeginningOfSentence">Indicate whether to include the beginning of sentence token in the encoding.</param>
        /// <param name="addEndOfSentence">Indicate whether to include the end of sentence token in the encoding.</param>
        /// <param name="unknownToken">The unknown token.</param>
        /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
        /// <param name="endOfSentenceToken">The end of sentence token.</param>
        internal CodeGenTokenizer(
                string vocabularyPath,
                string mergePath,
                PreTokenizer? preTokenizer = null,
                Normalizer? normalizer = null,
                IReadOnlyDictionary<string, int>? specialTokens = null,
                bool addPrefixSpace = false,
                bool addBeginningOfSentence = false,
                bool addEndOfSentence = false,
                string? unknownToken = DefaultSpecialToken,
                string? beginningOfSentenceToken = DefaultSpecialToken,
                string? endOfSentenceToken = DefaultSpecialToken) :
            this(vocabularyPath is null ? throw new ArgumentNullException(nameof(vocabularyPath)) : File.OpenRead(vocabularyPath),
                mergePath is null ? throw new ArgumentNullException(nameof(mergePath)) : File.OpenRead(mergePath),
                preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: true)
        {
        }

        /// <summary>
        /// Construct tokenizer's model object to use with the English Robert model.
        /// </summary>
        /// <param name="vocabularyStream">The stream of a JSON file containing the dictionary of string keys and their ids.</param>
        /// <param name="mergeStream">The stream of a file containing the tokens's pairs list.</param>
        /// <param name="preTokenizer">The pre-tokenizer to use.</param>
        /// <param name="normalizer">The normalizer to use.</param>
        /// <param name="specialTokens">The dictionary mapping special tokens to Ids.</param>
        /// <param name="addPrefixSpace">Indicate whether to include a leading space before encoding the text.</param>
        /// <param name="addBeginningOfSentence">Indicate whether to include the beginning of sentence token in the encoding.</param>
        /// <param name="addEndOfSentence">Indicate whether to include the end of sentence token in the encoding.</param>
        /// <param name="unknownToken">The unknown token.</param>
        /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass a valid non-null path to the model's vocab.json as the first argument.
2. Coalesce configuration with a fallback default path before calling the constructor.
3. Use the Stream-based constructor if you load the vocabulary from an embedded resource.

Example fix

// before
var tokenizer = new CodeGenTokenizer(config["VocabPath"], config["MergesPath"]);
// after
string vocabPath = config["VocabPath"] ?? Path.Combine(AppContext.BaseDirectory, "vocab.json");
string mergesPath = config["MergesPath"] ?? Path.Combine(AppContext.BaseDirectory, "merges.txt");
var tokenizer = new CodeGenTokenizer(vocabPath, mergesPath);

if (vocabularyPath is null || !File.Exists(vocabularyPath)) throw new ArgumentException("vocabularyPath must be non-null and exist");

if (vocabularyPath is string p && p.Length > 0) { /* safe */ }

try { var tok = new CodeGenTokenizer(vocabPath, mergesPath); } catch (ArgumentNullException ex) { logger.LogError(ex, "Null tokenizer path argument: {Param}", ex.ParamName); }

When it happens

Trigger: Calling the path-based CodeGenTokenizer constructor with a null vocabularyPath, e.g. new CodeGenTokenizer(null, "merges.txt") or passing an uninitialized config value.

Common situations: Config key for the vocab path missing so the bound value is null; caller believed the parameter was optional because many others have defaults.

Related errors

• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabulary));
• throw new ArgumentNullException(nameof(vocabFile));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/d052ac610188ae64. Report an issue: GitHub.