hyperledger/fabric · warning

unexpected end of blockfile

Error message

unexpected end of blockfile

What it means

ErrUnexpectedEndOfBlockfile is a sentinel error in Hyperledger Fabric's block storage indicating that a blockfile segment ends before a complete block could be read. It is thrown when the varint-encoded block length cannot be decoded from the trailing bytes, or when the declared length exceeds the bytes remaining in the file. This mainly happens if a crash occurs while a block was being appended, leaving partial block contents at the end of the file. Callers treat it as an expected condition for torn writes and truncate/recover rather than fail hard.

Source

Thrown at common/ledger/blkstorage/block_stream.go:22

SPDX-License-Identifier: Apache-2.0
*/

package blkstorage

import (
	"bufio"
	"fmt"
	"io"
	"os"

	"github.com/pkg/errors"
	"google.golang.org/protobuf/encoding/protowire"
)

// ErrUnexpectedEndOfBlockfile error used to indicate an unexpected end of a file segment
// this can happen mainly if a crash occurs during appending a block and partial block contents
// get written towards the end of the file
var ErrUnexpectedEndOfBlockfile = errors.New("unexpected end of blockfile")

// blockfileStream reads blocks sequentially from a single file.
// It starts from the given offset and can traverse till the end of the file
type blockfileStream struct {
	fileNum       int
	file          *os.File
	reader        *bufio.Reader
	currentOffset int64
}

// blockStream reads blocks sequentially from multiple files.
// it starts from a given file offset and continues with the next
// file segment until the end of the last segment (`endFileNum`)
type blockStream struct {
	rootDir           string
	currentFileNum    int
	endFileNum        int
	currentFileStream *blockfileStream

View on GitHub (pinned to 2736b63f8f)

Solutions

  1. Let the ledger recovery path handle it: it is an expected sentinel — allow scanForLastCompleteBlock/recovery to truncate the partial tail at the last complete block boundary
  2. Check disk space and fsync/journal health on the peer's ledger filesystem to prevent torn appends
  3. Do not copy or rsync blockfiles while the peer is running; stop the peer first
  4. If it occurs on a healthy file at offsets other than the end, validate the file with the blkstorage integrity tooling and restore from backup/recompute from a peer snapshot

Example fix

// before: treating any error from iteration as fatal
if err := stream.Err(); err != nil {
    return err // crashes recovery even for torn tail
}
// after
if errors.Is(err, blkstorage.ErrUnexpectedEndOfBlockfile) {
    logger.Warningf("partial block at file end, truncating")
    return truncateAfterLastCompleteBlock(file)
}
Defensive patterns

Strategy: type-guard

Validate before calling

info, err := os.Stat(blockfilePath)
if err != nil || info.Size() == 0 {
    // nothing to read or unreadable file; skip scan of this segment
}

Type guard

func isUnexpectedEOF(err error) bool {
    return errors.Is(err, blkstorage.ErrUnexpectedEndOfBlockfile)
}

Try / catch

if err := iter.Next(); err != nil {
    if isUnexpectedEOF(err) {
        logger.Warningf("partial block at tail; recovery will truncate")
        return recoverFromTornWrite()
    }
    return err
}

Prevention

When it happens

Trigger: Calling nextBlockBytesAndPlacementInfo (via nextBlockBytes) when: (1) the last read of the file hits fewer bytes than the varint length prefix needs and no more content is available (block_stream.go:113), or (2) lenBytes + length exceeds remainingBytes in the file (block_stream.go:120). Also deliberately returned by test helper testBlockFileStreamUnexpectedEOF.

Common situations: Peer crash or power loss mid-append leaving a partially written block; disk-full during block commit; process killed (SIGKILL) while flush is in flight; scanning a blockfile copied while the peer was still writing to it.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/52883a04ee974104. Report an issue: GitHub.