nats-io/nats-server · error

failed to read message sequence %d: unexpected payload size

Error message

failed to read message sequence %d: unexpected payload size

What it means

During RestoreStreamV2, each message block read from the backup has a declared size (declaredSize from the header record). After reading the block, the total bytes read must equal that declared size. This error means the actual payload length differs from the declared size, so the backup data is inconsistent or truncated.

Source

Thrown at server/stream_backup.go:510

			return nil, fmt.Errorf("snapshot message bytes exceed reserved restore size")
		}
		storedSize := int64(storedSizeRaw)
		if additional := storedSize - restoreRemaining; additional > 0 {
			jsa.updateUsage(tier, cfg.Storage, additional)
			restoreRemaining += additional
			if err := checkUsageLimits(); err != nil {
				return nil, err
			}
		}
		buf, err := io.ReadAll(tr)
		if err != nil {
			return nil, fmt.Errorf("failed to read message sequence %d: %w", seq, err)
		}
		if hdr.HeaderSize > int64(len(buf)) {
			return nil, fmt.Errorf("failed to parse message sequence %d: invalid header length", seq)
		}
		if int64(len(buf)) != declaredSize {
			return nil, fmt.Errorf("failed to read message sequence %d: unexpected payload size", seq)
		}
		subj := hdr.Name
		mhdr := buf[:hdr.HeaderSize]
		msg := buf[hdr.HeaderSize : hdr.HeaderSize+hdr.PayloadSize]
		if seq <= lseq {
			return nil, fmt.Errorf("message sequence %d out of order", seq)
		}
		// We could have deleted messages since the last message we stored, if so
		// we should work out what the gap is and skip those sequences.
		if gap := seq - lseq - 1; gap > 0 {
			if err := store.SkipMsgs(lseq+1, gap); err != nil {
				return nil, fmt.Errorf("failed to process gap: %w", err)
			}
		}
		lseq = seq
		ttl, err := getMessageTTL(mhdr)
		if err != nil {
			return nil, fmt.Errorf("failed to parse message TTL: %w", err)

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Re-create the backup and verify file size/checksums before restoring
2. Ensure the full backup file was transferred (compare byte counts between source and destination)
3. Do not manually edit or concatenate backup files; restore from the original archive
4. Use a compatible nats-server version for both backup (StreamBackup/backup API) and restore

if wantSize > 0 && gotSize != wantSize { return fmt.Errorf("backup size mismatch: got %d want %d", gotSize, wantSize) }

if err := restore(...); err != nil && strings.Contains(err.Error(), "unexpected payload size") {
    // abort and re-fetch a fresh backup
}

When it happens

Trigger: io.ReadAll(tr) returned a buffer whose length (len(buf)) does not equal the size declared in the block header for that sequence; occurs when the block stream is truncated, records were misaligned by earlier corruption, or the backup was produced/modified incorrectly.

Common situations: Restoring a partially downloaded backup, a backup whose file offsets shifted due to manual edits or merge attempts, or restoring a file written by a newer/older server version with a different block layout.

Related errors

• failed to parse message sequence %d: invalid header length
• message sequence %d out of order
• backup was truncated
• rebuildState for block %d failed: %w
• bad index file

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/c31c4e6712784262. Report an issue: GitHub.