dotnet/machinelearning · error · InvalidDataException

Malformed varint.

Error message

Malformed varint.

What it means

Thrown at the end of ReadRawVarint32 when ten consecutive bytes all have the continuation bit set, exceeding the maximum width of a protobuf varint. This means the byte stream is not valid protobuf — the SentencePiece reader refuses to continue rather than producing an undefined value.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:63

                    return result;
                }
            }

            // Negative int32 values are sign-extended to 10-byte varints; consume remaining bytes.
            for (int i = 0; i < 5; i++)
            {
                if (pos >= end)
                {
                    throw new InvalidDataException("Unexpected end of data while reading varint.");
                }

                if ((data[pos++] & 0x80) == 0)
                {
                    return result;
                }
            }

            throw new InvalidDataException("Malformed varint.");
        }

        internal static int ReadLengthPrefix(byte[] data, int end, ref int pos)
        {
            int length = ReadRawVarint32(data, end, ref pos);
            if ((uint)length > (uint)(end - pos))
            {
                throw new InvalidDataException("Invalid length-delimited field size.");
            }

            return length;
        }

        internal static string ReadString(byte[] data, int end, ref int pos)
        {
            int length = ReadLengthPrefix(data, end, ref pos);
            string result = Encoding.UTF8.GetString(data, pos, length);
            pos += length;

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Confirm the file really is a serialized SentencePiece ModelProto — open the first bytes and check for plausible protobuf framing, not text/HTML.
  2. Re-export the model from the original source with SentencePiece/protobuf tooling instead of hand-editing or re-encoding it.
  3. Verify the file was not transferred in text mode (FTP ASCII transfer or charset conversion corrupts binaries).
  4. If desynchronization is suspected after an earlier error, restart parsing from offset 0 of a validated buffer rather than continuing.

Example fix

// before
// vocab.json mistakenly passed to the SentencePiece tokenizer
var tokenizer = SentencePieceTokenizer.Create(File.ReadAllBytes("vocab.json"));

// after
if (!IsSentencePieceModel(modelPath)) // check magic/framing or correct file
    throw new InvalidOperationException($"'{modelPath}' is not a SentencePiece .model protobuf file.");
var tokenizer = SentencePieceTokenizer.Create(File.ReadAllBytes("sentencepiece.bpe.model"));
Defensive patterns

Strategy: validation

Validate before calling

// Ensure the file is binary protobuf, not text/HTML/JSON:
bool LooksBinary(byte[] d) => d.Length > 0 && (d[0] == 0x0A || char.IsControl((char)d[0]) == false && d[0] < 0x80 && d[0] != (byte)'<');
if (!LooksBinary(modelBytes))
    throw new InvalidDataException("Input does not look like a SentencePiece protobuf model.");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new InvalidOperationException("Data is not valid SentencePiece protobuf (malformed varint). Confirm you are loading the serialized .model file, not a text vocab or error page.", ex); }

Prevention

When it happens

Trigger: ReadRawVarint32 (reached via `length`) consumes bytes at shifts 0..28 and then the negative-value loop (5 more bytes, i == 4 without terminator), totalling more than the protobuf-spec max of 10 bytes, all with bit 0x80 set. Happens when the data is not actually protobuf or is desynchronized mid-stream.

Common situations: Passing a non-protobuf file (e.g. a raw text vocab, a BPE JSON file, or an HTML error page saved as .model) to the SentencePiece tokenizer; parsing a stream whose field framing got desynchronized earlier; binary corruption such as wrong encoding conversion (UTF-8 re-encode of a binary file).

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/31c0e1e32f03d55c. Report an issue: GitHub.