dotnet/machinelearning · error · InvalidOperationException

Invalid merger file format at line: {lineNumber}

Error message

Invalid merger file format at line: {lineNumber}

What it means

When loading a BPE merges file, each non-header line must contain exactly one space separating the two parts of the merge pair, with non-empty content on both sides. A line with no space, a trailing-only space, or more than one space is a malformed merges file and throws InvalidOperationException with the offending line number.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:1167

                string? line = useAsync ?
                    await Helpers.ReadLineAsync(reader, cancellationToken).ConfigureAwait(false) :
                    reader.ReadLine();

                if (line is null)
                {
                    break;
                }

                lineNumber++;
                if (line.StartsWith("#version", StringComparison.Ordinal) || line.Length == 0)
                {
                    continue;
                }

                int index = line.IndexOf(' ');
                if (index < 0 || index == line.Length - 1 || line.IndexOf(' ', index + 1) >= 0)
                {
                    throw new InvalidOperationException($"Invalid merger file format at line: {lineNumber}");
                }
                merges.Push((line.Substring(0, index), line.Substring(index + 1)));
            }

            return merges;
        }

        private readonly Dictionary<char, string> _charToString = new Dictionary<char, string>();

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        internal string CharToString(char c)
        {
            if (_charToString.TryGetValue(c, out string? v))
            {
                return v;
            }

            string s = c.ToString();

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the merges file is the plain-text HuggingFace merges.txt format: exactly two space-separated tokens per line.
2. Re-export or re-download the merges file for the specific model.
3. Check file encoding/line endings (re-save as UTF-8, LF) if the file was transferred across platforms.

Example fix

// before
var tokenizer = BpeTokenizer.Create(vocabPath, mergesPath); // merges.txt is actually a JSON array
// after
// ensure merges.txt lines look like: "t h" per line
var tokenizer = BpeTokenizer.Create(vocabPath, correctMergesPath);

foreach (var (line, i) in File.ReadLines(mergesPath).Select((l, i) => (l, i)))
{
    int sp = line.IndexOf(' ');
    if (sp < 0 || sp == line.Length - 1 || line.IndexOf(' ', sp + 1) >= 0)
        throw new InvalidDataException($"Bad merges line {i + 1}: '{line}'");
}

try { var tok = BpeTokenizer.Create(vocabPath, mergesPath); }
catch (InvalidOperationException ex) when (ex.Message.StartsWith("Invalid merger file format")) { /* re-export or fix merges file */ }

When it happens

Trigger: Loading a merges file where a line lacks the required single-space separator: a JSON merges array exported as text, a CRLF/encoding-corrupted file, or a merges file from a different tokenizer format.

Common situations: Pointing BPETokenizer at a vocab.json with the wrong merges.txt, hand-editing merges.txt, or using merges files saved by tools that write double-space entries.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• The max token count must be greater than 0.
• Unknown Token Out Of Vocabulary.
• Invalid merge file format at line: {lineNumber}
• Couldn't identify line breaks. Provided file is not text?
• {nameof(ChannelMessageKind)}.{e.Kind} is not yet implemented

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/7a6efd5036967a7b. Report an issue: GitHub.