dotnet/machinelearning · error · InvalidDataException

Unexpected end of data while skipping fixed32.

Error message

Unexpected end of data while skipping fixed32.

What it means

Thrown by SkipField for wire type 5 (32-bit fixed): fewer than 4 bytes remain before `end`, so the fixed32 value of an unknown field cannot be skipped. Like the other SkipField checks, it prevents out-of-bounds reads and flags the SentencePiece model stream as truncated.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:152

                    break;

                case 1: // 64-bit fixed
                    if (pos > end - 8)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed64.");
                    }
                    pos += 8;
                    break;

                case 2: // length-delimited
                    int skipLength = ReadLengthPrefix(data, end, ref pos);
                    pos += skipLength;
                    break;

                case 5: // 32-bit fixed
                    if (pos > end - 4)
                    {
                        throw new InvalidDataException("Unexpected end of data while skipping fixed32.");
                    }
                    pos += 4;
                    break;

                default:
                    throw new InvalidDataException($"Unknown or unsupported protobuf wire type {wireType}.");
            }
        }
    }

    /// <summary>Lightweight replacement for Google.Protobuf.ByteString with a Span property.</summary>
    internal readonly struct SentencePieceByteString(byte[] data, int offset, int length)
    {
        internal ReadOnlySpan<byte> Span => data is null ? ReadOnlySpan<byte>.Empty : data.AsSpan(offset, length);
    }

    /// <summary>ModelProto  (top-level message; field numbers match sentencepiece_model.proto)</summary>
    internal sealed class ModelProto

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Re-download/re-copy the model file and verify checksum; incomplete transfers are the usual cause.
  2. Clear any model cache and re-fetch (HuggingFace cache, app-local cache, etc.).
  3. Confirm the file is a raw SentencePiece ModelProto, not compressed or wrapped (tar/zip member extracted incorrectly).
  4. Wrap tokenizer creation in try-catch (InvalidDataException) with a retry that re-downloads the model.

Example fix

// before
try { tokenizer = SentencePieceTokenizer.Create(cacheBytes); }
catch (InvalidDataException) { throw; } // dead end

// after
try
{
    tokenizer = SentencePieceTokenizer.Create(cacheBytes);
}
catch (InvalidDataException ex)
{
    Log.Warn("Tokenizer model corrupt, re-downloading", ex);
    cacheBytes = DownloadModel(modelUrl);
    SaveToCache(modelKey, cacheBytes);
    tokenizer = SentencePieceTokenizer.Create(cacheBytes);
}
Defensive patterns

Strategy: try-catch

Validate before calling

bool IsCompleteDownload(byte[] bytes, long expectedLength) => bytes != null && bytes.LongLength == expectedLength;
if (!IsCompleteDownload(modelBytes, expectedLength))
    throw new InvalidDataException("Incomplete model download; refusing to parse.");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{
    PurgeCache(modelKey);
    modelBytes = DownloadModel(modelUrl);
    tokenizer = SentencePieceTokenizer.Create(modelBytes);
}

Prevention

When it happens

Trigger: SkipField encounters an unknown field with wire type 5 and evaluates pos > end - 4. Occurs when a truncated or corrupted model ends inside (or right at the start of) a fixed32 field, or after cursor desynchronization misreads a header as wire type 5.

Common situations: Download interrupted within the last bytes of the model (fixed32 fields such as float scores' unknown counterparts often sit late in the file); files copied incompletely (e.g. interrupted scp/rsync); damaged cache files.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6156aa5f0eb77c98. Report an issue: GitHub.