dotnet/machinelearning · error · IOException

Cannot read the file Merge file.{Environment.NewLine}Error m

Error message

Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}

What it means

Thrown when any exception occurs while reading and parsing the BPE merges.txt data file during CodeGenTokenizer construction. The library wraps all underlying failures (I/O, format, encoding) as IOException with this message so callers get a consistent exception type, preserving the original exception as InnerException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1796

                }

                int rank = 1;
                while (reader.Peek() >= 0)
                {
                    string line = reader.ReadLine()!;
                    int index = line.IndexOf(' ');
                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)
                    {
                        throw new FormatException($"Invalid format of merge file at line: \"{line}\"");
                    }

                    mergeRanks.Add(new StringSpanOrdinalKeyPair(line.Substring(0, index), line.Substring(index + 1)), rank++);
                }
            }
            catch (Exception e)
            {
                // Report any issues encountered while consuming a data file as IOExceptions.
                throw new IOException($"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}", e);
            }

            return mergeRanks;
        }

        private struct SymbolPair : IEquatable<SymbolPair>, IComparable<SymbolPair>
        {
            public int Left { get; set; }
            public int Right { get; set; }
            public int Length { get; set; }
            public int Score { get; set; }

            public SymbolPair(int left, int right, int score, int length)
            {
                Left = left;
                Right = right;
                Score = score;
                Length = length;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the merge file exists at the expected path and the stream is open, readable, and not already consumed (seek to 0 if reusable).
2. Check the InnerException to identify the root cause (FileNotFound, UnauthorizedAccess, decoder failure, etc.).
3. Validate the merge file content: each non-empty line must contain exactly one space separating two tokens with no leading space.
4. Copy vocab/merge data files with the application build (ensure they are included as content and deployed).

Example fix

// before
var tokenizer = new CodeGenTokenizer(vocabStream, mergeStream);
// after
if (!mergeStream.CanRead) throw new InvalidOperationException("Merge stream is not readable");
mergeStream.Seek(0, SeekOrigin.Begin);
try { var tokenizer = new CodeGenTokenizer(vocabStream, mergeStream); }
catch (IOException ex) { Console.WriteLine($"Merge file issue: {ex.InnerException?.Message}"); throw; }

if (!mergeStream.CanRead) throw new InvalidOperationException("Merge stream must be readable");
if (mergeStream.CanSeek) mergeStream.Seek(0, SeekOrigin.Begin);

try { var t = new CodeGenTokenizer(vocabStream, mergeStream); }
catch (IOException ex) { log(ex.InnerException ?? ex); throw new ApplicationException("Invalid or unreadable merge file", ex); }

When it happens

Trigger: Constructing a CodeGenTokenizer whose merge file stream is unreadable, a closed/disposed stream, a stream in the wrong encoding, or a stream whose content cannot be parsed line-by-line into 'token token' merge pairs.

Common situations: Passing a FileStream to a missing/locked file via a stream already consumed elsewhere; shipping a truncated or corrupted merges.txt with an app; opening the merge file with the wrong text encoding; a deployment step that did not copy the vocab data files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Cannot read the file Merge file.{Environment.NewLine}Error m
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• The special token '{kvp.Key}' is not in the vocabulary or as

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/a41a202d26b71f89. Report an issue: GitHub.