dotnet/machinelearning · error · InvalidOperationException

Invalid merger file format

Error message

Invalid merger file format

What it means

Each entry in BpeOptions.Merges must contain exactly one space separating the two parts of the merge pair ('left right'). Entries with no space, a trailing space (empty second part), or multiple spaces throw InvalidOperationException('Invalid merger file format'). This mirrors the merges.txt two-column format.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:179

                throw new InvalidOperationException("The vocabulary cannot be empty.");
            }

            Vec<(string, string)> merges = default;
            if (options.Merges is not null)
            {
                merges = new Vec<(string, string)>(1000);

                foreach (string merge in options.Merges)
                {
                    if (merge is null)
                    {
                        throw new InvalidOperationException("The merge entries cannot be null.");
                    }

                    int index = merge.IndexOf(' ');
                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)
                    {
                        throw new InvalidOperationException($"Invalid merger file format");
                    }

                    merges.Push((merge.Substring(0, index), merge.Substring(index + 1)));
                }
            }

            return new BpeTokenizer(
                            vocab, merges,
                            options.PreTokenizer,
                            options.Normalizer,
                            options.SpecialTokens,
                            options.UnknownToken,
                            options.ContinuingSubwordPrefix,
                            options.EndOfWordSuffix,
                            options.FuseUnknownTokens,
                            options.ByteLevel,
                            options.BeginningOfSentenceToken,
                            options.EndOfSentenceToken);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Normalize each merge line to exactly 'token1 token2' with a single ASCII space before Create.
2. Fix the source merges.txt to the standard BPE two-token-per-line format.
3. If your delimiter is a tab, convert: line.Replace('\t', ' ') and re-validate.
4. Pre-validate entries with a regex like ^\S+ \S+$ and reject/log bad lines before constructing the tokenizer.

Example fix

// before
options.Merges = File.ReadAllLines("merges.tsv"); // tab-separated -> invalid format
// after
options.Merges = File.ReadAllLines("merges.tsv")
    .Where(l => !string.IsNullOrWhiteSpace(l))
    .Select(l => l.Replace('\t', ' '))
    .Where(l => System.Text.RegularExpressions.Regex.IsMatch(l, @"^\S+ \S+$"))
    .ToArray();

var bad = merges?.Where(m => m is null || !System.Text.RegularExpressions.Regex.IsMatch(m, @"^\S+ \S+$")).ToList();
if (bad?.Count > 0) throw new FormatException($"Invalid merge entries: {string.Join('|', bad)}");

static bool IsValidMerge(string? m) => m is not null && System.Text.RegularExpressions.Regex.IsMatch(m, @"^\S+ \S+$");

try { var t = BpeTokenizer.Create(options); }
catch (InvalidOperationException ex) when (ex.Message.Contains("Invalid merger file format"))
{ /* log offending lines, repair delimiter, retry */ }

When it happens

Trigger: Passing merge strings like 'ab' (no space), 'ab ' (trailing space => empty second token), or 'a b c' (two spaces) in BpeOptions.Merges — e.g. from hand-edited merges files or lines with tab separators instead of spaces.

Common situations: Merges file saved with tabs/CSV formatting; version mismatch where merges use a different delimiter; copy-pasted merges losing or duplicating spaces; Windows line endings leaving stray characters (may also trip this depending on parsing).

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The vocabulary cannot be empty.
• The merge entries cannot be null.
• Invalid format of merge file at line: "{line}"

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/8695b687f854aad6. Report an issue: GitHub.