dotnet/machinelearning · error · InvalidDataException

Unexpected end of data while reading varint.

Error message

Unexpected end of data while reading varint.

What it means

This InvalidDataException is thrown by SentencePieceProtobufReader.ReadRawVarint32 when the reader position is already at or past the end of the data buffer before the first varint byte can be read. The library throws it while parsing a SentencePiece tokenizer model file (Protobuf wire format) to signal the byte stream is truncated or the framing is wrong, rather than returning garbage.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:24

// Replaces a full Google.Protobuf dependency with just enough wire-format reading
// to parse the fields the tokenizer implementation actually consumes.
// SentencePiece is under the Apache License 2.0 https://github.com/google/sentencepiece/blob/master/LICENSE

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace Sentencepiece
{
    /// <summary>Low-level protobuf wire-format primitives (read-only, forward-only).</summary>
    internal static class SentencePieceProtobufReader
    {
        internal static int ReadRawVarint32(byte[] data, int end, ref int pos)
        {
            if (pos >= end)
            {
                throw new InvalidDataException("Unexpected end of data while reading varint.");
            }

            byte b = data[pos++];
            int result = b & 0x7F;
            if ((b & 0x80) == 0)
            {
                return result;
            }

            for (int shift = 7; shift < 32; shift += 7)
            {
                if (pos >= end)
                {
                    throw new InvalidDataException("Unexpected end of data while reading varint.");
                }

                b = data[pos++];
                result |= (b & 0x7F) << shift;

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Re-download or re-copy the tokenizer model file and verify its byte size/checksum against the source.
  2. Check that the full serialized SentencePiece model (e.g. sentencepiece.bpe.model) is being passed, not a fragment — log data.Length before parsing.
  3. If the model is embedded as a resource, rebuild/re-embed it and confirm the embedded resource length matches the original file.
  4. Wrap model loading in a try-catch for InvalidDataException and surface a clear 'corrupt or truncated tokenizer model' message to the user.

Example fix

// before
byte[] modelData = File.ReadAllBytes(partialDownloadPath);
var tokenizer = SentencePieceTokenizer.Create(modelData);

// after
byte[] modelData = File.ReadAllBytes(modelPath);
if (modelData.Length == 0 || !VerifyChecksum(modelData, expectedSha256))
{
    throw new InvalidDataException($"SentencePiece model at '{modelPath}' is truncated or corrupt; re-download it.");
}
var tokenizer = SentencePieceTokenizer.Create(modelData);
Defensive patterns

Strategy: validation

Validate before calling

if (modelBytes == null || modelBytes.Length == 0)
    throw new InvalidDataException("SentencePiece model is empty; re-download it.");
// checksum check where a known hash exists:
using var sha = System.Security.Cryptography.SHA256.Create();
string hash = Convert.ToHexString(sha.ComputeHash(modelBytes));
if (hash != expectedHash)
    throw new InvalidDataException("SentencePiece model failed integrity check (truncated or corrupt).");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new ApplicationException("Tokenizer model is truncated or corrupt; re-download the .model file.", ex); }

Prevention

When it happens

Trigger: Calling ReadRawVarint32 (e.g. via ReadLengthPrefix/`length`) when `pos >= end`, i.e. the varint field begins exactly at the end of the buffer. Produced when a field header claims a field exists but there are zero bytes remaining — typically a truncated model file or a length prefix that overruns the buffer.

Common situations: Loading a truncated sentencepiece.bpe.model (incomplete download or HTTP stream cut off); a model file that was corrupted or partially written; passing a byte array containing only part of the serialized ModelProto to the tokenizer constructor; embedding a model resource that was mis-sized during build/packaging.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/48e6a6d87d095242. Report an issue: GitHub.