dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(mergePath));

Error message

throw new ArgumentNullException(nameof(mergePath));

What it means

The path-based CodeGenTokenizer constructor also requires mergePath and throws ArgumentNullException from the initializer when it is null. Both files (vocab and merges) are mandatory for this overload because the BPE model needs the merge rules. The null check happens before File.OpenRead so the error names the argument clearly.

Solutions

1. Pass a valid non-null path to the model's merges.txt as the second argument.
2. Ensure the merges asset is present and its path resolved before construction.
3. Prefer the Stream-based constructor with explicitly opened streams if you control loading.

Example fix

// before
var tokenizer = new CodeGenTokenizer(vocabPath, null);
// after
string mergesPath = Path.Combine(AppContext.BaseDirectory, "merges.txt");
var tokenizer = new CodeGenTokenizer(vocabPath, File.Exists(mergesPath) ? mergesPath : throw new FileNotFoundException(mergesPath));

if (mergePath is null || !File.Exists(mergePath)) throw new ArgumentException("mergePath must be non-null and exist");

if (mergePath is string p && p.Length > 0) { /* safe */ }

try { var tok = new CodeGenTokenizer(vocabPath, mergePath); } catch (ArgumentNullException ex) { logger.LogError(ex, "Null tokenizer path argument: {Param}", ex.ParamName); }

When it happens

Trigger: new CodeGenTokenizer("vocab.json", null) or mergePath resolved to null from missing configuration; assuming mergePath is optional like it is in BpeOptions.

Common situations: Porting code from BpeOptions (where mergesFile is optional) to CodeGenTokenizer; merges file deployment forgotten so the config value stays null.

Related errors

• throw new ArgumentNullException(nameof(vocabFile));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabulary));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/75bf4c317a486289. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:68

        /// <param name="addBeginningOfSentence">Indicate whether to include the beginning of sentence token in the encoding.</param>
        /// <param name="addEndOfSentence">Indicate whether to include the end of sentence token in the encoding.</param>
        /// <param name="unknownToken">The unknown token.</param>
        /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
        /// <param name="endOfSentenceToken">The end of sentence token.</param>
        internal CodeGenTokenizer(
                string vocabularyPath,
                string mergePath,
                PreTokenizer? preTokenizer = null,
                Normalizer? normalizer = null,
                IReadOnlyDictionary<string, int>? specialTokens = null,
                bool addPrefixSpace = false,
                bool addBeginningOfSentence = false,
                bool addEndOfSentence = false,
                string? unknownToken = DefaultSpecialToken,
                string? beginningOfSentenceToken = DefaultSpecialToken,
                string? endOfSentenceToken = DefaultSpecialToken) :
            this(vocabularyPath is null ? throw new ArgumentNullException(nameof(vocabularyPath)) : File.OpenRead(vocabularyPath),
                mergePath is null ? throw new ArgumentNullException(nameof(mergePath)) : File.OpenRead(mergePath),
                preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: true)
        {
        }

        /// <summary>
        /// Construct tokenizer's model object to use with the English Robert model.
        /// </summary>
        /// <param name="vocabularyStream">The stream of a JSON file containing the dictionary of string keys and their ids.</param>
        /// <param name="mergeStream">The stream of a file containing the tokens's pairs list.</param>
        /// <param name="preTokenizer">The pre-tokenizer to use.</param>
        /// <param name="normalizer">The normalizer to use.</param>
        /// <param name="specialTokens">The dictionary mapping special tokens to Ids.</param>
        /// <param name="addPrefixSpace">Indicate whether to include a leading space before encoding the text.</param>
        /// <param name="addBeginningOfSentence">Indicate whether to include the beginning of sentence token in the encoding.</param>
        /// <param name="addEndOfSentence">Indicate whether to include the end of sentence token in the encoding.</param>
        /// <param name="unknownToken">The unknown token.</param>
        /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
        /// <param name="endOfSentenceToken">The end of sentence token.</param>

View on GitHub (pinned to 7b76e69cf9)