dotnet/machinelearning · error · InvalidDataException

Unexpected end of data while skipping varint.

Error message

Unexpected end of data while skipping varint.

What it means

Thrown by SkipField when handling wire type 0 (varint): a continuation byte is required (high bit set) but pos >= end, so the unknown field's varint cannot be fully skipped. The library must consume the whole value to stay synchronized with the protobuf stream; it throws when the data ends too early.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:121

                buffer[2] = data[pos + 1];
                buffer[3] = data[pos];
                value = BitConverter.ToSingle(buffer, 0);
            }

            pos += 4;
            return value;
        }

        internal static void SkipField(byte[] data, int end, int wireType, ref int pos)
        {
            switch (wireType)
            {
                case 0: // varint (max 10 bytes per protobuf spec)
                    for (int i = 0; i < 10; i++)
                    {
                        if (pos >= end)
                        {
                            throw new InvalidDataException("Unexpected end of data while skipping varint.");
                        }

                        if ((data[pos++] & 0x80) == 0)
                        {
                            break;
                        }

                        if (i == 9)
                        {
                            throw new InvalidDataException("Malformed varint.");
                        }
                    }
                    break;

                case 1: // 64-bit fixed
                    if (pos > end - 8)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed64.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Re-download/verify the complete model file — the parse cannot recover from a mid-field truncation.
  2. Regenerate the model with a compatible SentencePiece version if unknown-field handling matters to you.
  3. Ensure the whole file is read into memory before parsing (no partial reads).
  4. Add a checksum check before tokenizer construction to detect truncation early.

Example fix

// before
var tokenizer = SentencePieceTokenizer.Create(truncatedStreamBytes);

// after
byte[] bytes = ReadAllBytesFully(stream); // loop until EOF
if (!ChecksumMatches(bytes))
    throw new InvalidDataException("Tokenizer model incomplete; re-download.");
var tokenizer = SentencePieceTokenizer.Create(bytes);
Defensive patterns

Strategy: try-catch

Validate before calling

// Detect truncation cheaply before loading:
if (modelBytes.Length == 0 || modelBytes[^1] == 0x80) // ends mid-continuation byte
    throw new InvalidDataException("Model appears truncated (ends on a varint continuation byte).");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new InvalidDataException("Could not skip an unknown field: model stream is truncated or from an incompatible protobuf version.", ex); }

Prevention

When it happens

Trigger: SkipField encounters an unknown field with wire type 0, iterates up to 10 varint bytes, and hits the end of the buffer before a byte with the continuation bit clear. Triggered by truncated models containing fields (e.g. newer ModelProto fields like trainer_spec variants) that must be skipped.

Common situations: Model serialized by a newer SentencePiece/protobuf version includes unknown fields; the (truncated) file ends inside such a field; partial downloads landing mid-unknown-field; binary corruption extending a varint past the buffer.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/146dc298c722fa62. Report an issue: GitHub.