microsoft/garnet · error · GarnetException

Malformed ids, {idLen} + {sizeof(int)} > {remainingIds.Lengt

Error message

Malformed ids, {idLen} + {sizeof(int)} > {remainingIds.Length}

What it means

Thrown by FetchVectorElementAttributes while iterating the length-prefixed id buffer returned by the DiskANN search service. Each entry is expected to be a 4-byte little-endian length followed by that many id bytes; if the declared idLen plus the 4-byte prefix exceeds the remaining buffer length, the buffer is considered malformed. Critically, this buffer (outputIds) is produced internally by the vector service, not supplied by the RESP client, so the throw signals an internal invariant violation or on-disk data corruption rather than bad client input.

Source

Thrown at libs/server/Resp/Vector/VectorManager.cs:1178

            byte[] idWithNamespaceArr = null;

            var attributesNextIx = 0;

            Span<byte> attributeFull = stackalloc byte[32];
            var attributeMem = SpanByteAndMemory.FromPinnedSpan(attributeFull);

            try
            {
                Span<byte> idWithNamespace = stackalloc byte[128];

                // TODO: we could scatter/gather this like MGET - doesn't matter when everything is in memory,
                //       but if anything is on disk it'd help perf
                for (var i = 0; i < numIds; i++)
                {
                    var idLen = BinaryPrimitives.ReadInt32LittleEndian(remainingIds);
                    if (idLen + sizeof(int) > remainingIds.Length)
                    {
                        throw new GarnetException($"Malformed ids, {idLen} + {sizeof(int)} > {remainingIds.Length}");
                    }

                    var id = remainingIds.Slice(sizeof(int), idLen);

                    // Make sure we've got enough space to query the element
                    if (id.Length + 1 > idWithNamespace.Length)
                    {
                        if (idWithNamespaceArr != null)
                        {
                            idPin.Free();
                            ArrayPool<byte>.Shared.Return(idWithNamespaceArr);
                        }

                        idWithNamespaceArr = ArrayPool<byte>.Shared.Rent(id.Length + 1);
                        idPin = GCHandle.Alloc(idWithNamespaceArr, GCHandleType.Pinned);
                        idWithNamespace = idWithNamespaceArr;
                    }

View on GitHub (pinned to 951b0fc683)

Solutions

1. Treat this as a likely data-corruption / internal-bug condition: capture the Vector Set key, the query, and the logs, and file an upstream issue.
2. Recreate the affected Vector Set (VDROP and rebuild via VADD) to eliminate corrupt on-disk state.
3. If it reproduces deterministically with a given dataset, try a different quantization (e.g. switch between Q8/BIN/FLOAT) to isolate a codec-specific bug.
4. Verify storage integrity: run with --recover from a known-good checkpoint and check the device/filesystem for errors.

try {
    await client.VSimilaritySearchAsync(key, queryVector, k, withAttrs: true);
} catch (GarnetException ex) when (ex.Message.StartsWith("Malformed ids")) {
    // Internal invariant / data corruption, not bad input. Do not retry unchanged.
    logger.LogCritical(ex, "Vector Set {Key} appears corrupt (malformed id buffer)", key);
    // Optionally quarantine the key and rebuild it.
}

When it happens

Trigger: Any VSIMILARITY/VSEARCH-style call that requests attributes or filtering (includeAttributes || !filter.IsEmpty) and then calls FetchVectorElementAttributes at VectorManager.cs:940 or :1109, when the DiskANN Service.SearchElement produced an outputIds buffer whose declared count (numIds) does not match its serialized length-prefix layout.

Common situations: Corrupted or partially-written Vector Set data on disk (e.g. an unclean shutdown or a truncated hybrid log), a serialization/deserialization bug in the DiskANN service for a specific vector dimension/quantization, or memory pressure causing a truncated output buffer. It is not caused by malformed RESP arguments.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Maximum Vector Set allocations exceeded, cannot issue new co
• VectorSetReplayTaskCount should be in range [0,{Environment.
• VectorSetQuantizationTaskCount should be in range [0,{Enviro
• MutablePercent must be between 10 and 95
• Store Log Memory size or PageCount must be specified

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/4c64e3d233c2a7ba. Report an issue: GitHub.