dotnet/machinelearning · error · InvalidOperationException

The merge value '{mergeValues.b}' is too short to be merged

Error message

The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.

What it means

When processing a merge whose right token should be split against a continuing-subword prefix length, the right token is shorter than (or equal to) the expected prefix length, so the split is impossible. The BpeTokenizer constructor throws InvalidOperationException, treating this as evidence that the merges file is damaged or inconsistent with the configured continuingSubwordPrefix.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:380

            Merges = new();
            for (int i = 0; i < merges.Count; i++)
            {
                (string a, string b) mergeValues = merges[i];

                if (!_vocab.TryGetValue(mergeValues.a, out int aId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.");
                }

                if (!_vocab.TryGetValue(mergeValues.b, out int bId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.");
                }

                if (mergeValues.b.Length <= prefixLen)
                {
                    throw new InvalidOperationException($"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.");
                }

                string newToken = $"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}";
                if (!_vocab.TryGetValue(newToken, out int newId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{newToken}' which not exist in the vocabulary.");
                }

                Merges.Add(new Pair<int>(aId, bId), (i, newId));
            }
        }

        /// <summary>
        /// Gets a value indicating whether to handle the input text in byte level.
        /// if true, the input text will be converted to UTF-8 bytes before encoding it.
        /// Additionally, some ASCII characters will be transformed to another characters (e.g Space character will be transformed to 'Ġ' character).
        /// </summary>
        public bool ByteLevel { get; }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass continuingSubwordPrefix: null (or the correct value) so it matches the actual prefix convention used in merges.txt.
2. Use the merges file belonging to the same model as the vocab — verify checksums.
3. Inspect the failing merge line and fix or remove the malformed entry.
4. Convert the merges file to the expected convention if migrating between tokenizers.

Example fix

// before
var tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: "Ġ"); // merges lack 'Ġ' prefixes
// after
var tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: null); // or use GPT-2-style merges

if (!string.IsNullOrEmpty(prefix))
    foreach (var (_, b) in merges)
        if (b.Length <= prefix.Length) throw new InvalidDataException($"Merge token '{b}' is shorter than prefix '{prefix}'; merges file does not match prefix convention");

null

try { var tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: prefix); }
catch (InvalidOperationException ex) when (ex.Message.Contains("too short to be merged")) { throw new InvalidDataException("continuingSubwordPrefix does not match merges.txt convention", ex); }

When it happens

Trigger: Constructing BpeTokenizer with a non-empty continuingSubwordPrefix (e.g. "Ġ") while the merges file contains right-hand tokens shorter than the prefix length, i.e. token strings that do not start with the expected prefix.

Common situations: Using merges.txt without the byte-level space marker while configuring continuingSubwordPrefix: "Ġ"; mixing models that use different space-prefix conventions (SentencePiece ▁ vs GPT-2 Ġ); corrupted merge lines.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Trying to merge a token '{mergeValues.a}' which not exist in
• Trying to merge a token '{mergeValues.b}' which not exist in
• Trying to merge a token '{newToken}' which not exist in the
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cedd01259d612fc3. Report an issue: GitHub.