dotnet/machinelearning · error · FormatException

Invalid format of merge file at line: "{line}"

Error message

Invalid format of merge file at line: "{line}"

What it means

Thrown when a line of the BPE merge file does not have the required format 'token1 token2'. A valid line must contain exactly one space, not at the start, not at the end. The library throws FormatException for this malformed line.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:219

            var mergeRanks = new Cache<(string, string), int>(60_000);
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file
                if (reader.Peek() >= 0)
                {
                    string ignored = reader.ReadLine()!;
                }

                int rank = 1;
                while (reader.Peek() >= 0)
                {
                    string line = reader.ReadLine()!;
                    int index = line.IndexOf(' ');
                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)
                    {
                        throw new FormatException($"Invalid format of merge file at line: \"{line}\"");
                    }

                    mergeRanks.Set((line.Substring(0, index), line.Substring(index + 1)), rank++);
                }
            }
            catch (Exception e)
            {
                // Report any issues encountered while consuming a data file as IOExceptions.
                throw new IOException($"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}", e);
            }

            return mergeRanks;
        }

        private Dictionary<string, int> GetVocab()
        {
            Dictionary<string, int>? publicVocab = Volatile.Read(ref _vocabOriginal);
            if (publicVocab is null)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Open the merge file and inspect the reported line; fix it to contain exactly two space-separated tokens with no leading/trailing space.
2. Re-download the canonical merges.txt for the model instead of hand-editing.
3. Pre-validate the file: reject lines where IndexOf(' ') < 1, is last char, or a second space exists.
4. Check for encoding issues (BOM, non-breaking spaces) and re-save as UTF-8 without BOM.

Example fix

// before (merges.txt line)
"  Ġ t"
// after
"Ġ t"

foreach (var (line, i) in File.ReadLines(mergesPath).Select((l, i) => (l, i)))
{
    int sp = line.IndexOf(' ');
    bool ok = sp >= 1 && sp < line.Length - 1 && line.IndexOf(' ', sp + 1) == -1;
    if (!ok && line.Length > 0) throw new InvalidDataException($"Bad merge line {i + 1}: '{line}'");
}

try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }
catch (FormatException ex) { throw new InvalidDataException($"merges.txt malformed: {ex.Message}", ex); }

When it happens

Trigger: A merge file line with no space, with a leading space, with a trailing space, or with two or more spaces (three or more tokens) is read by GetMergeRanks during EnglishRobertaTokenizer construction.

Common situations: Hand-edited merges.txt; a file converted between encodings that altered whitespace; using a merges file from an incompatible tokenizer version with different line format; CRLF/BOM corruption from copy-paste.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• The special token '{kvp.Key}' is not in the vocabulary or as
• The special token '{token}' is not in the vocabulary.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/4bd44011edd8e945. Report an issue: GitHub.