apache/pulsar · error · IOException

Fail to read LedgerMetadata for ledgerId ${key}

Error message

Fail to read LedgerMetadata for ledgerId ${key}

What it means

When opening a FileStoreBackedReadHandle, the offloaded ledger's index entry is read from the filesystem and deserialized with parseLedgerMetadata. If that fails — missing/corrupt metadata file, unreadable bytes — the constructor logs and rethrows an IOException naming the ledgerId, leaving the handle unopened.

Source

Thrown at tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileStoreBackedReadHandleImpl.java:85

        this.ledgerId = ledgerId;
        this.executor = executor;
        this.reader = reader;
        this.offloaderStats = offloaderStats;
        this.managedLedgerName = managedLedgerName;
        this.topicName = TopicName.fromPersistenceNamingEncoding(managedLedgerName);
        LongWritable key = new LongWritable();
        BytesWritable value = new BytesWritable();
        try {
            key.set(FileSystemManagedLedgerOffloader.METADATA_KEY_INDEX);
            long startReadIndexTime = System.nanoTime();
            reader.get(key, value);
            offloaderStats.recordReadOffloadIndexLatency(topicName,
                    System.nanoTime() - startReadIndexTime, TimeUnit.NANOSECONDS);
            this.ledgerMetadata = parseLedgerMetadata(ledgerId, value.copyBytes());
            state = State.Opened;
        } catch (IOException e) {
            log.error().attr("ledgerId", ledgerId).log("Fail to read LedgerMetadata");
            throw new IOException("Fail to read LedgerMetadata for ledgerId " + key.get());
        }
    }

    @Override
    public long getId() {
        return ledgerId;
    }

    @Override
    public LedgerMetadata getLedgerMetadata() {
        return ledgerMetadata;

    }

    @Override
    public CompletableFuture<Void> closeAsync() {
        if (closeFuture.get() != null || !closeFuture.compareAndSet(null, new CompletableFuture<>())) {
            return closeFuture.get();

View on GitHub (pinned to 820761864e)

Solutions

1. Verify the offload location (fileSystemUri/profile config) still points to the directory holding the offloaded ledger's index file
2. Check the index file exists and is intact for the reported ledgerId; re-offload the segment from the source ledger or restore from backup
3. Fix filesystem permissions for the broker user running tiered-storage reads
4. Re-run offload for the affected segment if the stored LedgerMetadata is corrupt/unparseable

Example fix

// config pointing at wrong store
offloader: fileSystemUri: hdfs://old-cluster:9000
// after
offloader: fileSystemUri: hdfs://new-cluster:9000  (path restored/verified)

// verify the offload index exists before reading
Path idx = Paths.get(offloadBaseDir, "index", ledgerId + ".index");
if (!Files.isRegularFile(idx) || Files.size(idx) == 0) {
  throw new IllegalStateException("offload index missing for ledger " + ledgerId);
}

try {
  ReadHandle rh = FileStoreBackedReadHandleImpl.get(...);
} catch (IOException e) {
  if (e.getMessage().startsWith("Fail to read LedgerMetadata")) {
    log.error("Offload index corrupt/missing for ledger — restore from backup or re-offload");
  } else throw e;
}

When it happens

Trigger: constructReadHandle fetches the index blob for ledgerId from the FileStore offload location and parseLedgerMetadata throws; thrown as new IOException("Fail to read LedgerMetadata for ledgerId " + key.get()).

Common situations: Offloaded data directory moved/deleted or bucket path misconfigured; partially-written or truncated index from an interrupted offload; wrong filesystem/table config (fileSystemProfile/uri) pointing at the wrong store; permissions on the offload directory.

Related errors

• Error seeking, new position %d < current position %d
• java.io.IOException (wraps InterruptedException | ExecutionE
• Data block header magic word not match. read: ${magic} expec
• Cursor %s mark-delete position %s is ahead of the last posit
• Bookie 'group' parameters is missing

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/114187b5fdebfe24. Report an issue: GitHub.