nats-io/nats-server · critical

rebuildState for block %d failed: %w

Error message

rebuildState for block %d failed: %w

What it means

Emitted during stream recovery when a message block's rebuildState fails — the store could not reconstruct the block's in-memory index (message offsets, sequence mappings) from the file. The block is skipped and the error recorded via storeErr, so the stream recovers with potentially missing messages from that block.

Source

Thrown at server/filestore.go:7542

	}

	if err := fs.checkAndFlushLastBlock(); err != nil {
		return nil, storeErr(fmt.Errorf("flush of last block failed: %w", err))
	}

	// Clear any global subject state.
	fs.psim, fs.tsl = fs.psim.Empty(), 0

	for _, mb := range fs.blks {
		// Make sure encryption loaded if needed for the block.
		if err := fs.loadEncryptionForMsgBlock(mb); err != nil {
			_ = storeErr(fmt.Errorf("loading encryption for block %d failed: %w", mb.index, err))
			continue
		}
		// FIXME(dlc) - check tombstones here too?
		ld, _, err := mb.rebuildState()
		if err != nil {
			_ = storeErr(fmt.Errorf("rebuildState for block %d failed: %w", mb.index, err))
			continue
		}
		if ld != nil {
			// Rebuild fs state too.
			fs.rebuildStateLocked(ld)
		}
		if err = fs.populateGlobalPerSubjectInfo(mb); err != nil {
			_ = storeErr(fmt.Errorf("populating per-subject info for block %d failed: %w", mb.index, err))
			continue
		}
	}

	return fs.ld, firstErr
}

// Lock should be held.
func (mb *msgBlock) enableForWriting(fip bool) error {
	if mb == nil {

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Read the wrapped cause in server logs — it pinpoints the block index and failure
2. Restart the server; JetStream may recover other blocks and mark this one bad
3. If block data is expendable, remove that block's .blk/.fst files so recovery skips it
4. Check storage health (fsck, SMART) and re-restore the stream from a backup
5. Enable/disable filestore sync options and upgrade NATS — rebuildState has been hardened over releases

Example fix

// skip a corrupt block so the stream recovers with earlier data
rm /path/jetstream/$ACC/streams/ORDERS/msgblk/7.blk
rm /path/jetstream/$ACC/streams/ORDERS/msgblk/7.fst
# then restart nats-server

// Go: confirm stream state matches expectations after any recovery error
info, err := js.StreamInfo(ctx, "ORDERS")
if err == nil && info.State.FirstSeq > expectedFirstSeq {
    log.Printf("blocks were skipped during recovery: first seq now %d", info.State.FirstSeq)
}

if err != nil {
    if strings.Contains(err.Error(), "rebuildState for block") {
        // partial recovery: a block was skipped
        // 1. capture server logs  2. restore from backup or accept data loss
        return reconcileAfterBlockLoss(js, "ORDERS")
    }
    return err
}

When it happens

Trigger: Recover() iterating fs.blks where mb.rebuildState() returns an error: truncated or partially written .blk file, corrupt per-message metadata, mismatch between file size and recorded write offsets after a crash.

Common situations: Power loss / kill -9 without clean shutdown leaving a partially flushed block; disk corruption; manually editing or copying incomplete stream files; storage layer (NFS) losing writes.

Related errors

• loading encryption for block %d failed: %w
• populating per-subject info for block %d failed: %w
• error opening msg block file [%q]: %v
• bad index file
• could not decode avl dmap: %v

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/4851bf6f06d4827f. Report an issue: GitHub.