dotnet/machinelearning · error · InvalidOperationException

Invalid merge file format at line: {lineNumber}

Error message

Invalid merge file format at line: {lineNumber}

What it means

BpeOptions validates each non-comment line of the merges file must contain exactly one space separating two tokens, with a non-empty second token and no additional spaces. Any line violating this (zero spaces, trailing space, or two+ spaces) throws InvalidOperationException with the offending line number. This guards against a corrupt or wrong-format merges file.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs:86

                List<string> merges = new();

                int lineNumber = 0;
                string? line;

                while ((line = reader.ReadLine()) is not null)
                {
                    lineNumber++;
                    if (line.StartsWith("#version", StringComparison.Ordinal) || line.Length == 0)
                    {
                        continue;
                    }

                    // validate the merges format
                    int index = line.IndexOf(' ');
                    if (index < 0 || index == line.Length - 1 || line.IndexOf(' ', index + 1) >= 0)
                    {
                        throw new InvalidOperationException($"Invalid merge file format at line: {lineNumber}");
                    }

                    merges.Add(line);
                }

                Merges = merges;
            }
        }

        /// <summary>
        /// Gets or sets the vocabulary to use.
        /// </summary>
        public IEnumerable<KeyValuePair<string, int>> Vocabulary { get; }

        /// <summary>
        /// Gets or sets the list of the merge strings used to merge tokens during encoding.
        /// </summary>
        public IEnumerable<string>? Merges { get; set; }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Open merges.txt at the reported line number and fix the line to exactly 'token1 token2' with a single space.
2. Re-download the merges file from the original model repository instead of a hand-edited copy.
3. Strip BOM and normalize line endings (LF) if the file was re-encoded.
4. Pre-validate all lines with a regex like ^\S+ \S+$ before constructing BpeOptions.

Example fix

// before (merges.txt line 42: 'l  o')
var options = new BpeOptions(vocabPath, mergesPath);
// after: validate lines first
var bad = File.ReadAllLines(mergesPath)
    .Select((line, i) => (line, i))
    .Where(x => !string.IsNullOrWhiteSpace(x.line) && !x.line.StartsWith("#") && !System.Text.RegularExpressions.Regex.IsMatch(x.line, @"^\S+ \S+$"))
    .ToList();
if (bad.Count > 0) throw new InvalidDataException($"Bad merges line {bad[0].i + 1}");
var options = new BpeOptions(vocabPath, mergesPath);

var bad = File.ReadAllLines(mergesPath)
    .Select((line, i) => (line, i))
    .Where(x => !string.IsNullOrWhiteSpace(x.line) && !x.line.StartsWith("#") && !System.Text.RegularExpressions.Regex.IsMatch(x.line, @"^\S+ \S+$"))
    .ToList();
if (bad.Count > 0) throw new InvalidDataException($"Malformed merges line(s): {string.Join(",", bad.Select(b => b.i + 1))}");

try { var options = new BpeOptions(vocabPath, mergesPath); } catch (InvalidOperationException ex) { logger.LogError(ex, "Malformed merges file {Path}", mergesPath); }

When it happens

Trigger: merges.txt contains a malformed line such as 'a b' (double space), 'ab' (no space), 'a ' (trailing space), or CRLF/encoding artifacts; passing a non-merges text file (e.g. a plain word list) as mergesFile.

Common situations: Manual edits to the merges file introduced stray whitespace; file re-encoded with BOM or converted line endings; using merges from an incompatible tokenizer version.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• throw new ArgumentNullException(nameof(vocabStream));
• Invalid merger file format at line: {lineNumber}
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• The special token '{kvp.Key}' is not in the vocabulary or as

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/b4d501758c0ed38e. Report an issue: GitHub.