dotnet/machinelearning · error · InvalidOperationException

The merge entries cannot be null.

Error message

The merge entries cannot be null.

What it means

When BpeOptions.Merges is supplied, each merge entry must be a non-null string; a null entry throws InvalidOperationException('The merge entries cannot be null.'). Merges define the BPE pair-merge table and must all be valid.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:173

            {
                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);
            }

            if (vocab.Count == 0)
            {
                throw new InvalidOperationException("The vocabulary cannot be empty.");
            }

            Vec<(string, string)> merges = default;
            if (options.Merges is not null)
            {
                merges = new Vec<(string, string)>(1000);

                foreach (string merge in options.Merges)
                {
                    if (merge is null)
                    {
                        throw new InvalidOperationException("The merge entries cannot be null.");
                    }

                    int index = merge.IndexOf(' ');
                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)
                    {
                        throw new InvalidOperationException($"Invalid merger file format");
                    }

                    merges.Push((merge.Substring(0, index), merge.Substring(index + 1)));
                }
            }

            return new BpeTokenizer(
                            vocab, merges,
                            options.PreTokenizer,
                            options.Normalizer,
                            options.SpecialTokens,
                            options.UnknownToken,

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Filter out null (and empty) entries before Create: merges.Where(m => m != null).
2. Fix the file/loader so blank or malformed lines are skipped rather than converted to null.
3. Validate with Array.FindIndex(merges, m => m == null) to find the offending index before calling Create.
4. If deserializing JSON, use a deserializer setting that ignores null items or sanitize afterward.

Example fix

// before
options.Merges = rawLines; // rawLines contains nulls from blank lines
// after
options.Merges = rawLines.Where(m => !string.IsNullOrEmpty(m)).ToArray();

if (merges?.Any(m => m is null) == true)
    throw new InvalidOperationException("Merges contains null entries.");

static bool AllMergesNonNull(string[]? m) => m is not null && Array.TrueForAll(m, x => x is not null);

try { var t = BpeTokenizer.Create(options); }
catch (InvalidOperationException ex) when (ex.Message.Contains("merge entries cannot be null"))
{ options.Merges = options.Merges.Where(x => x != null).ToArray(); var t = BpeTokenizer.Create(options); }

When it happens

Trigger: Passing a List<string> (or string[]) to BpeOptions.Merges that contains at least one null element — commonly from array initialization with unfilled slots, deserialization of sparse lists, or Select() producing nulls.

Common situations: Reading merge lines from a file that has trailing/blank lines mapped to null; constructing merges = new string[n] without filling all slots; LINQ projection returning null for malformed rows.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The vocabulary cannot be empty.
• Invalid merger file format
• Invalid format of merge file at line: "{line}"

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/00b02cdcfc2d1732. Report an issue: GitHub.