dotnet/machinelearning · error · InvalidDataException

Unknown or unsupported protobuf wire type {wireType}.

Error message

Unknown or unsupported protobuf wire type {wireType}.

What it means

This InvalidDataException is thrown by the hand-written protobuf SkipField helper in SentencepieceModel.cs when it encounters a wire type it does not know how to skip while reading a sentencepiece model protobuf. The library only supports the standard wire types (varint, fixed64, length-delimited, fixed32); anything else means the stream is corrupt or not actually a protobuf message. It acts as a guard against silently consuming a malformed model file.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:158

                    }
                    pos += 8;
                    break;

                case 2: // length-delimited
                    int skipLength = ReadLengthPrefix(data, end, ref pos);
                    pos += skipLength;
                    break;

                case 5: // 32-bit fixed
                    if (pos > end - 4)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed32.");
                    }
                    pos += 4;
                    break;

                default:
                    throw new InvalidDataException($"Unknown or unsupported protobuf wire type {wireType}.");
            }
        }
    }

    /// <summary>Lightweight replacement for Google.Protobuf.ByteString with a Span property.</summary>
    internal readonly struct SentencePieceByteString(byte[] data, int offset, int length)
    {
        internal ReadOnlySpan<byte> Span => data is null ? ReadOnlySpan<byte>.Empty : data.AsSpan(offset, length);
    }

    /// <summary>ModelProto  (top-level message; field numbers match sentencepiece_model.proto)</summary>
    internal sealed class ModelProto
    {
        internal static readonly ModelProtoParser Parser = new();

        internal List<Types.SentencePiece> Pieces { get; } = new();
        internal TrainerSpec TrainerSpec { get; private set; } = new();
        internal NormalizerSpec NormalizerSpec { get; private set; } = new();

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Re-download or regenerate the sentencepiece .model file and verify its integrity (checksum/size).
  2. Confirm the file being passed is actually a protobuf sentencepiece model (open it — it should start as compact binary, typically beginning with byte 0x0A for field 1).
  3. Check that no preprocessing step (transfer mode, encoding conversion, line-ending munging) is corrupting the file before parsing.
  4. If building from source text, regenerate with the same sentencepiece trainer version used by the library.

Example fix

// before: pointing at a wrong/HTML file downloaded from a URL
var tokenizer = SentencePieceTokenizer.Create(modelPath: "model.html");
// after: validate the file is a protobuf before loading
byte[] head = File.ReadAllBytes("model.html")[..2];
if (head[0] != 0x0A) throw new InvalidDataException("Not a sentencepiece protobuf model");
var tokenizer = SentencePieceTokenizer.Create(modelPath: "model.html");
Defensive patterns

Strategy: try-catch

Validate before calling

byte[] head = new byte[2];
using (var fs = File.OpenRead(modelPath)) { if (fs.Read(head) < 2) throw new InvalidDataException("Model file too small"); }
bool likelyProtobuf = head[0] == 0x0A;

Type guard

static bool IsLikelySentencepieceModel(byte[] data) => data is { Length: > 2 } && data[0] == 0x0A && data[1] > 0;

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelStream); }
catch (InvalidDataException ex) when (ex.Message.Contains("wire type"))
{ throw new ApplicationException("Model file is corrupt or not a protobuf sentencepiece model", ex); }

Prevention

When it happens

Trigger: Deserializing a sentencepiece .model file whose encoded tag byte contains a wire type other than 0, 1, 2, or 5 — e.g. the file is truncated and misaligned so a value byte is parsed as a tag, or the file is not a protobuf at all (HTML error page, wrong file downloaded).

Common situations: Downloading a corrupted or partial tokenizer model, pointing the tokenizer at the wrong file, a model serialized by an incompatible/newer protobuf schema using newer wire types (e.g. wire type 3/4 groups or 6+), or manually concatenating protobuf streams.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/c72d67754e00346c. Report an issue: GitHub.