LykosAI/StabilityMatrix · error · InvalidDataException

JSON length exceeds the maximum allowed size.

Error message

JSON length exceeds the maximum allowed size.

What it means

SafetensorMetadata.ParseAsync reads an 8-byte little-endian header and throws InvalidDataException when the declared JSON length exceeds 100 MB. This guards against corrupt or malicious safetensors files causing huge allocations.

Solutions

1. Re-download or re-export the safetensors file; the header is corrupt.
2. Verify the file is actually safetensors format (starts with plausible 8-byte length followed by '{').
3. Check that the stream position is 0 before parsing.
4. Validate file size and header before calling ParseAsync.

Example fix

// before
await SafetensorMetadata.ParseAsync(stream); // throws on corrupt header
// after
if (stream.Length < 8) throw new InvalidDataException("File too small for safetensors header.");
var header = new byte[8];
await stream.ReadExactlyAsync(header);
var len = BinaryPrimitives.ReadUInt64LittleEndian(header);
if (len > 100UL * 1024 * 1024) throw new InvalidDataException($"Unsafe header length {len}.");
stream.Position = 0;
await SafetensorMetadata.ParseAsync(stream);

const ulong MaxJsonLength = 100UL * 1024 * 1024;
using var view = await MemoryMappedFile.CreateFromFile(path).CreateViewStreamAsync(0, 8);
var buf = new byte[8];
await view.ReadExactlyAsync(buf);
var len = System.Buffers.Binary.BinaryPrimitives.ReadUInt64LittleEndian(buf);
if (len > MaxJsonLength) throw new InvalidDataException($"Header length {len} is unsafe.");

bool HasPlausibleSafetensorsHeader(ReadOnlySpan<byte> first9Bytes) =>
    first9Bytes.Length >= 9 &&
    System.Buffers.Binary.BinaryPrimitives.ReadUInt64LittleEndian(first9Bytes) <= 100UL * 1024 * 1024 &&
    first9Bytes[8] == (byte)'{';

try { var meta = await SafetensorMetadata.ParseAsync(stream); }
catch (InvalidDataException ex) { _logger.LogWarning(ex, "Rejecting file: {Message}", ex.Message); return null; }

When it happens

Trigger: Loading a .safetensors file whose first 8 bytes encode a JSON size greater than 100 MB — corrupt file, wrong offset, or a non-safetensors file renamed to .safetensors.

Common situations: Truncated or corrupted downloads, files with missing/shifted headers, malicious crafted files, reading the wrong region of a stream.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to deserialize metadata.
• Invalid metadata JSON.
• Invalid metadata JSON, expected property name but got
• JSON does not start with '{'.

AI-assisted analysis of LykosAI/StabilityMatrix@af93d6ef57 (2026-09-12). Data as JSON: /api/errors/81249d2978dd191d. Report an issue: GitHub.

Appendix: source

Thrown at StabilityMatrix.Core/Models/SafetensorMetadata.cs:44

    /// <summary>
    /// Tries to parse the metadata from a SafeTensor file.
    /// </summary>
    /// <param name="safetensorStream">Stream to the SafeTensor file.</param>
    /// <returns>The parsed metadata. Can be <see langword="null"/> if the file does not contain metadata.</returns>
    public static async Task<SafetensorMetadata?> ParseAsync(Stream safetensorStream)
    {
        // 8 bytes unsigned little-endian 64-bit integer
        // 1 byte start of JSON object '{'
        Memory<byte> buffer = new byte[9];
        await safetensorStream.ReadExactlyAsync(buffer).ConfigureAwait(false);
        var span = buffer.Span;

        const ulong MAX_ALLOWED_JSON_LENGTH = 100 * 1024 * 1024; // 100 MB
        var jsonLength = BinaryPrimitives.ReadUInt64LittleEndian(span);
        if (jsonLength > MAX_ALLOWED_JSON_LENGTH)
        {
            throw new InvalidDataException("JSON length exceeds the maximum allowed size.");
        }
        if (span[8] != '{')
        {
            throw new InvalidDataException("JSON does not start with '{'.");
        }

        // Unfornately Utf8JsonReader does not support reading from a stream directly.
        // Usually the size of the entire JSON object is less than 500KB,
        // using a pooled buffer should reduce the number of large allocations.
        var jsonBytes = ArrayPool<byte>.Shared.Rent((int)jsonLength);
        try
        {
            // Important: the length of the rented buffer can be larger than jsonLength
            // and there can be additional junk data at the end.

            // we already read {, so start from index 1
            jsonBytes[0] = (byte)'{';
            await safetensorStream

View on GitHub (pinned to af93d6ef57)