What causes "Unexpected end of data while reading float." in dotnet/machinelearning?

Re-download the model file and verify its checksum — truncation inside a float array is the dominant cause.

dotnet/machinelearning · error · InvalidDataException

Unexpected end of data while reading float.

Error message

Unexpected end of data while reading float.

What it means

Thrown by SentencePieceProtobufReader.ReadFloat when fewer than 4 bytes remain before `end`, so a 32-bit IEEE float field (e.g. sentencepiece piece `score`) cannot be read. The library checks explicitly to avoid reading past the buffer and to report the stream as truncated.

Source

Thrown at src/Microsoft.ML.Tokenizers/SentencepieceModel.cs:89

                throw new InvalidDataException("Invalid length-delimited field size.");
            }

            return length;
        }

        internal static string ReadString(byte[] data, int end, ref int pos)
        {
            int length = ReadLengthPrefix(data, end, ref pos);
            string result = Encoding.UTF8.GetString(data, pos, length);
            pos += length;
            return result;
        }

        internal static float ReadFloat(byte[] data, int end, ref int pos)
        {
            if (pos > end - 4)
            {
                throw new InvalidDataException("Unexpected end of data while reading float.");
            }

            float value;
            if (BitConverter.IsLittleEndian)
            {
                value = BitConverter.ToSingle(data, pos);
            }
            else
            {
                // Protobuf fixed32 is always little-endian; reverse bytes on big-endian platforms.
                byte[] buffer = new byte[4];
                buffer[0] = data[pos + 3];
                buffer[1] = data[pos + 2];
                buffer[2] = data[pos + 1];
                buffer[3] = data[pos];
                value = BitConverter.ToSingle(buffer, 0);
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

Re-download the model file and verify its checksum — truncation inside a float array is the dominant cause.
Reassemble streamed downloads correctly (use CopyTo/ReadLoop, not single Read calls) before parsing.
Validate the model with the official protobuf deserializer as a pre-check in your pipeline.
Fail fast at startup with checksum verification rather than deep inside tokenization.

Example fix

// before
byte[] buf = new byte[fileSize];
using var fs = File.OpenRead(path);
int read = fs.Read(buf, 0, buf.Length); // may read fewer bytes
var tokenizer = SentencePieceTokenizer.Create(buf);

// after
byte[] buf;
using (var ms = new MemoryStream())
{
    using var fs = File.OpenRead(path);
    fs.CopyTo(ms);
    buf = ms.ToArray();
}
var tokenizer = SentencePieceTokenizer.Create(buf);

Defensive patterns

Strategy: validation

Validate before calling

// Ensure the whole file was read before parsing:
byte[] ReadAllBytes(string path)
{
    using var fs = File.OpenRead(path);
    using var ms = new MemoryStream();
    fs.CopyTo(ms);
    var bytes = ms.ToArray();
    if (bytes.Length != new FileInfo(path).Length)
        throw new InvalidDataException("Incomplete read of tokenizer model.");
    return bytes;
}

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }
catch (InvalidDataException ex)
{ throw new InvalidDataException("Model stream truncated inside a float field — re-download and verify checksum.", ex); }

Prevention

Use CopyTo loops, never assume a single Read fills the buffer
Verify downloads against Content-Length and checksum before parsing
When memory-mapping/splicing buffers, double-check the end offset covers the full model
Load models once at startup, inside a try-catch with a re-download fallback

When it happens

Trigger: ReadFloat encounters pos > end - 4 — a float-typed field header was parsed but the remaining payload holds fewer than 4 bytes. Occurs when truncation lands inside a repeated score/array field of the ModelProto.

Common situations: Model file truncated inside a large repeated float array (common in sentencepiece models with thousands of scores); chunked HTTP download assembled with a dropped chunk; memory-mapped or span-based reads with a wrong window end.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/3e771987ea9c646a. Report an issue: GitHub.