dotnet/machinelearning · error · FormatException

Invalid format of merge file at line: "{line}"

Error message

Invalid format of merge file at line: "{line}"

What it means

GetMergeRanks parses the BPE merges file line by line, requiring each line to contain exactly one space separating a non-empty pair of parts. Any line violating this (no space, space at start/end, or multiple spaces) throws FormatException identifying the offending line.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1787

            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file
                if (reader.Peek() >= 0)
                {
                    string ignored = reader.ReadLine()!;
                }

                int rank = 1;
                while (reader.Peek() >= 0)
                {
                    string line = reader.ReadLine()!;
                    int index = line.IndexOf(' ');
                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)
                    {
                        throw new FormatException($"Invalid format of merge file at line: \"{line}\"");
                    }

                    mergeRanks.Add(new StringSpanOrdinalKeyPair(line.Substring(0, index), line.Substring(index + 1)), rank++);
                }
            }
            catch (Exception e)
            {
                // Report any issues encountered while consuming a data file as IOExceptions.
                throw new IOException($"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}", e);
            }

            return mergeRanks;
        }

        private struct SymbolPair : IEquatable<SymbolPair>, IComparable<SymbolPair>
        {
            public int Left { get; set; }
            public int Right { get; set; }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify you passed merges.txt (line-based merge pairs), not vocab.json, as the merge stream.
2. Open merges.txt and check each line has exactly two non-empty parts separated by a single space; fix or remove malformed lines.
3. Re-download the merges file for the exact model checkpoint (matching the vocabulary).
4. Preprocess the stream to strip blank lines/BOM before passing it to the tokenizer factory.

Example fix

// before
var tok = CodeGenTokenizer.Create(vocabStream, vocabStream); // vocab parsed as merges -> 'Invalid format of merge file'
// after
var tok = CodeGenTokenizer.Create(vocabStream, mergesStream);

foreach (var line in File.ReadLines(mergesPath))
{
    int i = line.IndexOf(' ');
    if (i < 1 || i == line.Length - 1 || line.IndexOf(' ', i + 1) != -1)
        throw new InvalidDataException($"Bad merge line: {line}");
}

bool IsValidMergeLine(string line) { int i = line.IndexOf(' '); return i > 0 && i < line.Length - 1 && line.IndexOf(' ', i + 1) == -1; }

try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }
catch (FormatException ex) when (ex.Message.StartsWith("Invalid format of merge file")) { /* repair or re-download merges.txt */ }

When it happens

Trigger: Supplying a merges file whose lines are not of the form 'tokenA tokenB' — e.g. the vocab JSON passed as merges, an empty first header line retained, blank lines, CRLF remnants, or a merges file from an incompatible tokenizer version.

Common situations: Swapped vocab/merges files, merges.txt with the '#version' header handling differences, files downloaded with wrong line endings, or hand-edited merges files.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Couldn't identify line breaks. Provided file is not text?
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The vocabulary cannot be empty.
• The merge entries cannot be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/21b9b86489bf856c. Report an issue: GitHub.