apache/pulsar · error · EOFException

Header was too small

Error message

Header was too small

What it means

After reading the fixed header fields, fromStream skips forward headerLen bytes to reach the payload. If the underlying stream cannot skip that many bytes (dis.skip returns a short count), the stream is shorter than the declared header length, so an EOFException('Header was too small') is thrown. This indicates a truncated or malformed data block.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockHeaderImpl.java:66

        return new DataBlockHeaderImpl(HEADER_MAX_SIZE, blockLength, firstEntryId);
    }

    // Construct DataBlockHeader from InputStream, which contains `HEADER_MAX_SIZE` bytes readable.
    public static DataBlockHeader fromStream(InputStream stream) throws IOException {
        CountingInputStream countingStream = new CountingInputStream(stream);
        DataInputStream dis = new DataInputStream(countingStream);
        int magic = dis.readInt();
        if (magic != MAGIC_WORD) {
            throw new IOException("Data block header magic word not match. read: " + magic
                    + " expected: " + MAGIC_WORD);
        }

        long headerLen = dis.readLong();
        long blockLen = dis.readLong();
        long firstEntryId = dis.readLong();
        long toSkip = headerLen - countingStream.getCount();
        if (dis.skip(toSkip) != toSkip) {
            throw new EOFException("Header was too small");
        }

        return new DataBlockHeaderImpl(headerLen, blockLen, firstEntryId);
    }

    private final long headerLength;
    private final long blockLength;
    private final long firstEntryId;

    public static int getBlockMagicWord() {
        return MAGIC_WORD;
    }

    public static int getDataStartOffset() {
        return HEADER_MAX_SIZE;
    }

    @Override

View on GitHub (pinned to 820761864e)

Solutions

1. Re-offload the ledger segment; the offloaded object is truncated and cannot be repaired in place
2. Verify the object's size in blob storage matches the expected block length and re-download if it differs
3. Check storage provider logs for incomplete multipart uploads or failed writes around the offload time
4. If reading over a network stream, buffer the full block locally before parsing the header

Example fix

// before: parsing directly from a streaming HTTP payload
DataBlockHeader h = DataBlockHeaderImpl.fromStream(httpResponse.getEntity().getContent());
// after: read fully into memory first so skip cannot fall short
byte[] all = ByteStreams.toByteArray(httpResponse.getEntity().getContent());
DataBlockHeader h = DataBlockHeaderImpl.fromStream(new ByteArrayInputStream(all));

// check the object is at least as big as the header claims
long objectSize = blob.getMetadata().getContentLength();
if (objectSize < DataBlockHeaderImpl.HEADER_MAX_SIZE) {
    throw new IOException("Offload block truncated: " + objectSize + " bytes");
}

try {
    DataBlockHeader h = DataBlockHeaderImpl.fromStream(stream);
} catch (EOFException e) {
    // stream shorter than declared headerLen: object truncated, re-offload
    log.warn("Offloaded data block truncated", e);
}

When it happens

Trigger: DataBlockHeaderImpl.fromStream is given a stream containing fewer bytes than the headerLength value declared in the block header itself — e.g. a partially uploaded object, a network read cut short, or a header whose headerLen field was corrupted.

Common situations: Interrupted offload writes leaving partial objects; storage provider truncating large blobs; manual copy of an offloaded block that stopped mid-transfer; restored backup missing trailing bytes.

Related errors

• Timeout during reset cursor
• Fail to read LedgerMetadata for ledgerId ${key}
• Error reading from BlobStore
• Error seeking, new position %d < current position %d
• java.io.IOException (wraps InterruptedException | ExecutionE

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/c73cf6db325cf16d. Report an issue: GitHub.