dotnet/machinelearning · error · IOException

Cannot read the file Merge file.

Error message

Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}

What it means

Thrown when any exception occurs while reading and parsing the BPE merges.txt data file during CodeGenTokenizer construction. The library wraps all underlying failures (I/O, format, encoding) as IOException with this message so callers get a consistent exception type, preserving the original exception as InnerException.

Solutions

1. Verify the merge file exists at the expected path and the stream is open, readable, and not already consumed (seek to 0 if reusable).
2. Check the InnerException to identify the root cause (FileNotFound, UnauthorizedAccess, decoder failure, etc.).
3. Validate the merge file content: each non-empty line must contain exactly one space separating two tokens with no leading space.
4. Copy vocab/merge data files with the application build (ensure they are included as content and deployed).

Example fix

// before
var tokenizer = new CodeGenTokenizer(vocabStream, mergeStream);
// after
if (!mergeStream.CanRead) throw new InvalidOperationException("Merge stream is not readable");
mergeStream.Seek(0, SeekOrigin.Begin);
try { var tokenizer = new CodeGenTokenizer(vocabStream, mergeStream); }
catch (IOException ex) { Console.WriteLine($"Merge file issue: {ex.InnerException?.Message}"); throw; }

if (!mergeStream.CanRead) throw new InvalidOperationException("Merge stream must be readable");
if (mergeStream.CanSeek) mergeStream.Seek(0, SeekOrigin.Begin);

try { var t = new CodeGenTokenizer(vocabStream, mergeStream); }
catch (IOException ex) { log(ex.InnerException ?? ex); throw new ApplicationException("Invalid or unreadable merge file", ex); }

When it happens

Trigger: Constructing a CodeGenTokenizer whose merge file stream is unreadable, a closed/disposed stream, a stream in the wrong encoding, or a stream whose content cannot be parsed line-by-line into 'token token' merge pairs.

Common situations: Passing a FileStream to a missing/locked file via a stream already consumed elsewhere; shipping a truncated or corrupted merges.txt with an app; opening the merge file with the wrong text encoding; a deployment step that did not copy the vocab data files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Cannot read the file Merge file.
• argument should not be null.
• Could not find the merges file
• Could not find the vocabulary file
• Failed to read the vocabulary file.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/a41a202d26b71f89. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1796

                }

                int rank = 1;
                while (reader.Peek() >= 0)
                {
                    string line = reader.ReadLine()!;
                    int index = line.IndexOf(' ');
                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)
                    {
                        throw new FormatException($"Invalid format of merge file at line: \"{line}\"");
                    }

                    mergeRanks.Add(new StringSpanOrdinalKeyPair(line.Substring(0, index), line.Substring(index + 1)), rank++);
                }
            }
            catch (Exception e)
            {
                // Report any issues encountered while consuming a data file as IOExceptions.
                throw new IOException($"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}", e);
            }

            return mergeRanks;
        }

        private struct SymbolPair : IEquatable<SymbolPair>, IComparable<SymbolPair>
        {
            public int Left { get; set; }
            public int Right { get; set; }
            public int Length { get; set; }
            public int Score { get; set; }

            public SymbolPair(int left, int right, int score, int length)
            {
                Left = left;
                Right = right;
                Score = score;
                Length = length;

View on GitHub (pinned to 7b76e69cf9)