microsoft/garnet · critical · TsavoriteException

Unexpected end of snapshot object-log data while copying obj

Error message

Unexpected end of snapshot object-log data while copying objects during recovery

What it means

Thrown by ObjectLogWriter.CopyRecoveredObjectBytes when copying object-log bytes from the snapshot object reader during recovery and reader.Read returns 0 (EOF) before the expected totalLength bytes are consumed. The snapshot object-log stream is shorter than the main-log record says it should be — the object-log snapshot is truncated or mismatched with the main log.

Source

Thrown at libs/storage/Tsavorite/cs/src/core/Allocator/ObjectSerialization/ObjectLogWriter.cs:147

        /// positioned at the record (via <see cref="CircularDiskReadBuffer.OnBeginRecord"/>).
        /// </summary>
        /// <param name="reader">The reader over the snapshot object-log, positioned at the record to copy.</param>
        /// <param name="totalLength">The total number of object-log bytes for the record (key plus value).</param>
        public void CopyRecoveredObjectBytes(ObjectLogReader<TStoreFunctions> reader, ulong totalLength)
        {
            if (totalLength > 0)
            {
                var buffer = flushBuffers.bufferPool.Get(IStreamBuffer.BufferSize);
                try
                {
                    var chunkSpan = buffer.TotalValidSpan;
                    var remaining = totalLength;
                    while (remaining > 0)
                    {
                        var requestLength = (int)Math.Min(remaining, (ulong)chunkSpan.Length);
                        var bytesRead = reader.Read(chunkSpan.Slice(0, requestLength));
                        if (bytesRead == 0)
                            throw new TsavoriteException("Unexpected end of snapshot object-log data while copying objects during recovery");
                        Write(chunkSpan.Slice(0, bytesRead));
                        remaining -= (ulong)bytesRead;
                    }
                }
                finally
                {
                    flushBuffers.bufferPool.Return(buffer);
                }
            }

            // Signal completion, as WriteRecordObjects does.
            flushBuffers.OnRecordComplete();
        }

        /// <summary>Start off the write using the full span of the <see cref="OverflowByteArray"/>.</summary>
        /// <param name="overflow">The <see cref="OverflowByteArray"/> to write.</param>
        void WriteDirect(OverflowByteArray overflow) => WriteDirect(overflow, overflow.ReadOnlySpan, refCountedGCHandle: default);

View on GitHub (pinned to 951b0fc683)

Solutions

1. Restore the complete object-log snapshot from the same checkpoint as the main-log snapshot.
2. Verify the object-log snapshot file size is at least as large as the highest referenced position before recovery.
3. Recover from an earlier consistent checkpoint if the object-log snapshot is unusable.
4. Ensure durable flush of the object-log device on checkpoint to avoid a short file after a crash.

Example fix

// before: recover with a truncated object-log snapshot
store.Recover();
// after: restore the full matching object-log snapshot device, then recover
logSettings.ObjectLogDevice = Devices.CreateLogDevice(fullObjectLogPath);
store.Recover();

// Verify the object-log snapshot length covers the highest referenced position before Recover
var fi = new FileInfo(objectLogSnapshotPath);
if (!fi.Exists || fi.Length < maxReferencedObjectPosition)
    throw new InvalidOperationException("Object-log snapshot missing or truncated");

try { store.Recover(); }
catch (TsavoriteException ex) when (ex.Message.Contains("Unexpected end of snapshot object-log data"))
{
    logger.LogError(ex, "Object-log snapshot truncated; restore complete checkpoint and retry");
    throw;
}

When it happens

Trigger: Recovery object-log copy where the snapshot object-log device ends prematurely relative to the copyObjectLength the main record references. Triggered in the recovery loop after CreateSnapshotObjectReader positions the reader.

Common situations: Truncated or partially restored object-log snapshot file; snapshot of object log taken from a different/older checkpoint than the main log; object-log device write that didn't flush fully before crash; corruption cutting the file short.

Related errors

• No snapshot object-log data available while copying objects
• Native handle is null.
• Snapshot path is required.
• Failed to take CPR snapshot of BfTree.
• recoveryPath is required.

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/241c9dddbb8e7bea. Report an issue: GitHub.