apache/pulsar · error · IllegalArgumentException

Cursor %s mark-delete position %s is ahead of the last posit

Error message

Cursor %s mark-delete position %s is ahead of the last position %s for managed ledger %s

What it means

StreamingDataBlockHeaderImpl.fromStream reads the leading 4-byte magic word from a data block header read back from tiered storage and throws this IOException when it does not equal the expected MAGIC_WORD. The library uses the magic word as a integrity/format marker for offloaded ledger data blocks. A mismatch means the stream is not positioned at a valid data block header or the stored data is corrupted, truncated, or written by an incompatible format version.

Source

Thrown at managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:1289

    @Override
    public int getNonContiguousDeletedMessagesRangeSerializedSize() {
        return this.individualDeletedMessagesSerializedSize;
    }

    @Override
    public long getEstimatedSizeSinceMarkDeletePosition() {
        Position markDeletePosition = this.markDeletePosition;
        Position lastPosition = ledger.getLastPosition();
        if (markDeletePosition == null || markDeletePosition.compareTo(lastPosition) == 0) {
            return 0;
        }
        if (markDeletePosition.compareTo(lastPosition) > 0) {
            if (!ledger.ledgerExists(lastPosition.getLedgerId())
                    || isMarkDeletePositionOnEmptyCurrentLedger(markDeletePosition)) {
                return 0;
            }
            throw new IllegalArgumentException(String.format(
                    "Cursor %s mark-delete position %s is ahead of the last position %s for managed ledger %s",
                    name, markDeletePosition, lastPosition, ledger.getName()));
        }

        long totalSize = ledger.estimateBacklogFromPosition(markDeletePosition);

        // Need to subtract size of individual deleted messages
        log.debug()
                .attr("markDeletePosition", markDeletePosition)
                .attr("totalSize", totalSize)
                .log("Calculating backlog size");

        // Get count of individually deleted entries in the backlog range
        long deletedCount = 0;
        lock.readLock().lock();
        try {
            Range<Position> backlogRange = Range.openClosed(markDeletePosition, lastPosition);
            deletedCount = individualDeletedMessages.cardinality(

View on GitHub (pinned to 820761864e)

Solutions

1. Verify the stream is positioned exactly at the start of a data block before calling fromStream (reset/reopen the InputStream at offset 0 of the block).
2. Re-offload the ledger to a new location — the stored blob is likely corrupted or truncated; do not attempt to hand-patch the magic word.
3. Check that the reader and writer use compatible BookKeeper/Pulsar versions for the streaming offload format.
4. Inspect the actual bytes read (first 4 bytes) against MAGIC_WORD to confirm whether it is an offset problem (valid magic elsewhere in the stream) vs corruption (garbage bytes).

Example fix

// before: stream may be positioned mid-block
InputStream in = blob.getPayload().openStream();
in.skip(someOffset);
StreamingDataBlockHeaderImpl.fromStream(in);

// after: ensure stream is at block start
try (InputStream in = blob.getPayload().openStream()) {
    StreamingDataBlockHeaderImpl.fromStream(in); // reads from offset 0
}

if (available(stream) < 4) { throw new IOException("Stream too short for data block header"); }

try {
    StreamingDataBlockHeader header = StreamingDataBlockHeaderImpl.fromStream(in);
} catch (IOException e) {
    if (e.getMessage().contains("magic word")) {
        log.warn("Corrupt or misaligned offloaded block header");
        // re-offload or fail the read-back, do not retry same stream
    }
    throw e;
}

When it happens

Trigger: Calling StreamingDataBlockHeaderImpl.fromStream(InputStream) on a stream whose first 4 bytes are not the expected MAGIC_WORD — e.g. the InputStream is positioned past the header, points at the wrong offset in the blob, the blob content is corrupted/overwritten, or the stream yields bytes in the wrong order (endianness/decoding issue).

Common situations: Reading an offloaded ledger object from object storage (S3/GCS/Azure) that was truncated by a failed upload, manually edited or re-uploaded, or read from the wrong blob/key; restoring with a BookKeeper/Pulsar version whose block format differs; buggy custom offload code that passes a mid-file stream instead of the block start.

Related errors

• Data block header magic word not match. read: ${magic} expec
• Timeout during reset cursor
• IOException
• IOException
• Fail to read LedgerMetadata for ledgerId ${key}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8bd39eabba99a0d6. Report an issue: GitHub.