What causes "error removing the block file [%s]" in hyperledger/fabric?

Stop all processes using the ledger directory (peer, containers) and re-run the rollback.

error removing the block file [%s]

Error message

error removing the block file [%s]

What it means

During rollbackBlockFiles, all block files with a number greater than the target file are removed so the ledger can be rolled back. If os.Remove fails on any of those files, the error is wrapped as "error removing the block file [%s]" and the rollback aborts, leaving files in a partially rolled-back state until retried.

Source

Thrown at common/ledger/blkstorage/rollback.go:192

		return err
	}
	// must not use index for block location search since the index can be behind the target block
	targetFileNum, err := binarySearchFileNumForBlock(r.ledgerDir, r.targetBlockNum)
	if err != nil {
		return err
	}
	lastFileNum, err := retrieveLastFileSuffix(r.ledgerDir)
	if err != nil {
		return err
	}

	logger.Infof("Removing all block files with suffixNum in the range [%d] to [%d]",
		targetFileNum+1, lastFileNum)

	for n := lastFileNum; n >= targetFileNum+1; n-- {
		filepath := deriveBlockfilePath(r.ledgerDir, n)
		if err := os.Remove(filepath); err != nil {
			return errors.Wrapf(err, "error removing the block file [%s]", filepath)
		}
	}

	logger.Infof("Truncating block file [%d] to the end boundary of block number [%d]", targetFileNum, r.targetBlockNum)
	endOffset, err := calculateEndOffSet(r.ledgerDir, targetFileNum, r.targetBlockNum)
	if err != nil {
		return err
	}

	filePath := deriveBlockfilePath(r.ledgerDir, targetFileNum)
	if err := os.Truncate(filePath, endOffset); err != nil {
		return errors.Wrapf(err, "error truncating the block file [%s]", filePath)
	}

	return nil
}

func calculateEndOffSet(ledgerDir string, targetBlkFileNum int, blockNum uint64) (int64, error) {

View on GitHub (pinned to 2736b63f8f)

Solutions

Stop all processes using the ledger directory (peer, containers) and re-run the rollback.
Check permissions/ownership of the ledger directory and files (chown/chmod to the peer user) and ensure the filesystem is writable.
Free disk space or remount the volume read-write if the FS reported EROFS/ENOSPC.
Remove immutable flags (chattr -i) if set, then retry rollback.

Example fix

// before
$ peer node rollback --channelID mychannel --blockNumber 50  // peer still running
// after
$ docker stop peer0.org1
$ peer node rollback --channelID mychannel --blockNumber 50

Defensive patterns

Strategy: try-catch

Validate before calling

for n := lastFileNum; n >= targetFileNum+1; n-- {
    p := deriveBlockfilePath(ledgerDir, n)
    if f, err := os.OpenFile(p, os.O_RDWR, 0); err != nil {
        return fmt.Errorf("cannot remove %s (locked or missing): %w", p, err)
    } else {
        f.Close()
    }
}

Try / catch

if err := os.Remove(filepath); err != nil {
    if errors.Is(err, os.ErrPermission) || errors.Is(err, syscall.EBUSY) {
        return fmt.Errorf("stop processes using %s and retry: %w", filepath, err)
    }
    return errors.Wrapf(err, "error removing the block file [%s]", filepath)
}

Prevention

Ensure the peer and all containers using the ledger dir are stopped before rollback.
Run rollback as the same OS user that owns the ledger files.
Ensure the ledger volume is mounted read-write with free disk space.
Avoid immutable attributes or external file locks on ledger files.

When it happens

Trigger: peer node rollback (RollbackKVLedger / rollbackBlockFiles) while the block file to delete is open by another process, is read-only, resides on a read-only/full filesystem, or the OS denies unlink permission on the ledger directory.

Common situations: Another peer process still running against the same ledger directory, root-owned files created by a different container run, read-only mounted volume during maintenance, or immutable file attributes (chattr +i).

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/e547869f74658a10. Report an issue: GitHub.