dotnet/machinelearning · error · InvalidDataException

Unexpected end of data while skipping fixed64.

Error message

Unexpected end of data while skipping fixed64.

What it means

Thrown by SkipField for wire type 1 (64-bit fixed): fewer than 8 bytes remain before `end`, so the fixed64 value of an unknown field cannot be skipped. Thrown to keep the protobuf cursor consistent — the parser refuses to read past the buffer edge.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:139

                            throw new InvalidDataException("Unexpected end of data while skipping varint.");
                        }

                        if ((data[pos++] & 0x80) == 0)
                        {
                            break;
                        }

                        if (i == 9)
                        {
                            throw new InvalidDataException("Malformed varint.");
                        }
                    }
                    break;

                case 1: // 64-bit fixed
                    if (pos > end - 8)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed64.");
                    }
                    pos += 8;
                    break;

                case 2: // length-delimited
                    int skipLength = ReadLengthPrefix(data, end, ref pos);
                    pos += skipLength;
                    break;

                case 5: // 32-bit fixed
                    if (pos > end - 4)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed32.");
                    }
                    pos += 4;
                    break;

                default:

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Re-download the model and verify byte-for-byte integrity (size + hash).
  2. Use a model exported with a SentencePiece/protobuf version compatible with your library to avoid unknown fixed64 fields.
  3. Ensure full-file reads (CopyTo loop) rather than single Read calls that may under-fill the buffer.
  4. Validate the model with an independent protobuf decoder before passing it to the tokenizer.

Example fix

// before
var tokenizer = SentencePieceTokenizer.Create(partialBytes); // ends mid-field

// after
byte[] bytes = File.ReadAllBytes(modelPath);
if (bytes.Length != expectedModelLength)
    throw new InvalidDataException($"Model file is {bytes.Length} bytes, expected {expectedModelLength}; re-download.");
var tokenizer = SentencePieceTokenizer.Create(bytes);
Defensive patterns

Strategy: try-catch

Validate before calling

if (modelBytes.Length != expectedSize)
    throw new InvalidDataException($"Model incomplete: {modelBytes.Length}/{expectedSize} bytes. Re-download before loading.");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new InvalidDataException("Unknown fixed64 field truncated: the model file is incomplete. Re-download it.", ex); }

Prevention

When it happens

Trigger: SkipField sees an unknown field with wire type 1 and evaluates pos > end - 8. Occurs when a truncated model ends inside a fixed64 field (e.g. double-valued unknown fields added by newer protobuf definitions).

Common situations: Model file cut off during download inside a fixed64 field; using a model saved by a newer SentencePiece version whose extra double fields the current parser must skip; corrupted length prefixes shifting the cursor into wrong field boundaries.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/b46cdedc07c0121e. Report an issue: GitHub.