dotnet/machinelearning · error · InvalidDataException

Invalid length-delimited field size.

Error message

Invalid length-delimited field size.

What it means

Thrown by ReadLengthPrefix when the varint length of a length-delimited field exceeds the number of bytes remaining in the buffer (length > end - pos). The library uses this check to ensure every string/bytes field (e.g. sentencepiece piece strings) lies within the buffer before reading it.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:71

                {
                    throw new InvalidDataException("Unexpected end of data while reading varint.");
                }

                if ((data[pos++] & 0x80) == 0)
                {
                    return result;
                }
            }

            throw new InvalidDataException("Malformed varint.");
        }

        internal static int ReadLengthPrefix(byte[] data, int end, ref int pos)
        {
            int length = ReadRawVarint32(data, end, ref pos);
            if ((uint)length > (uint)(end - pos))
            {
                throw new InvalidDataException("Invalid length-delimited field size.");
            }

            return length;
        }

        internal static string ReadString(byte[] data, int end, ref int pos)
        {
            int length = ReadLengthPrefix(data, end, ref pos);
            string result = Encoding.UTF8.GetString(data, pos, length);
            pos += length;
            return result;
        }

        internal static float ReadFloat(byte[] data, int end, ref int pos)
        {
            if (pos > end - 4)
            {
                throw new InvalidDataException("Unexpected end of data while reading float.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Verify and re-download the model file; compare byte length with a known-good copy.
  2. Check the code that produces the byte slice — ensure end offsets reflect the actual buffer length.
  3. If the model came from a package/cache, clear the cache (e.g. ~/.cache/huggingface) and let it re-fetch.
  4. Wrap tokenizer creation in try-catch (InvalidDataException) and fail with a 're-download model' message.

Example fix

// before
var tokenizer = SentencePieceTokenizer.Create(cachedBytes); // cache truncated

// after
byte[] cachedBytes = LoadFromCache(modelKey);
if (cachedBytes == null || cachedBytes.Length != knownGoodLength)
{
    cachedBytes = DownloadModel(modelUrl);
    SaveToCache(modelKey, cachedBytes);
}
var tokenizer = SentencePieceTokenizer.Create(cachedBytes);
Defensive patterns

Strategy: validation

Validate before calling

if (modelBytes == null || modelBytes.Length != expectedModelLength)
    throw new InvalidDataException($"Model size mismatch: got {modelBytes?.Length ?? 0} bytes, expected {expectedModelLength}. Re-download.");

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new InvalidDataException("Length-delimited field overruns buffer — model is truncated. Re-download the tokenizer model.", ex); }

Prevention

When it happens

Trigger: ReadLengthPrefix (called from `length` and `skipLength`) reads a length varint whose unsigned value is larger than the remaining byte count — a field header promising more payload than the buffer contains. Caused by truncation or a corrupted length varint.

Common situations: Downloading a model via a connection that dropped mid-transfer; an incomplete asset embedded during build; corrupt length varint from bit flips; slicing a shared buffer with the wrong end offset so the parser sees a length prefix near the buffer edge.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/3c03564e55519b7e. Report an issue: GitHub.