{"record":{"id":"114187b5fdebfe24","repo":"apache/pulsar","slug":"fail-to-read-ledgermetadata-for-ledgerid-key","errorCode":null,"errorMessage":"Fail to read LedgerMetadata for ledgerId ${key}","messagePattern":"Fail to read LedgerMetadata for ledgerId (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileStoreBackedReadHandleImpl.java","lineNumber":85,"sourceCode":"        this.ledgerId = ledgerId;\n        this.executor = executor;\n        this.reader = reader;\n        this.offloaderStats = offloaderStats;\n        this.managedLedgerName = managedLedgerName;\n        this.topicName = TopicName.fromPersistenceNamingEncoding(managedLedgerName);\n        LongWritable key = new LongWritable();\n        BytesWritable value = new BytesWritable();\n        try {\n            key.set(FileSystemManagedLedgerOffloader.METADATA_KEY_INDEX);\n            long startReadIndexTime = System.nanoTime();\n            reader.get(key, value);\n            offloaderStats.recordReadOffloadIndexLatency(topicName,\n                    System.nanoTime() - startReadIndexTime, TimeUnit.NANOSECONDS);\n            this.ledgerMetadata = parseLedgerMetadata(ledgerId, value.copyBytes());\n            state = State.Opened;\n        } catch (IOException e) {\n            log.error().attr(\"ledgerId\", ledgerId).log(\"Fail to read LedgerMetadata\");\n            throw new IOException(\"Fail to read LedgerMetadata for ledgerId \" + key.get());\n        }\n    }\n\n    @Override\n    public long getId() {\n        return ledgerId;\n    }\n\n    @Override\n    public LedgerMetadata getLedgerMetadata() {\n        return ledgerMetadata;\n\n    }\n\n    @Override\n    public CompletableFuture<Void> closeAsync() {\n        if (closeFuture.get() != null || !closeFuture.compareAndSet(null, new CompletableFuture<>())) {\n            return closeFuture.get();","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileStoreBackedReadHandleImpl.java#L67-L103","documentation":"When opening a FileStoreBackedReadHandle, the offloaded ledger's index entry is read from the filesystem and deserialized with parseLedgerMetadata. If that fails — missing/corrupt metadata file, unreadable bytes — the constructor logs and rethrows an IOException naming the ledgerId, leaving the handle unopened.","triggerScenarios":"constructReadHandle fetches the index blob for ledgerId from the FileStore offload location and parseLedgerMetadata throws; thrown as new IOException(\"Fail to read LedgerMetadata for ledgerId \" + key.get()).","commonSituations":"Offloaded data directory moved/deleted or bucket path misconfigured; partially-written or truncated index from an interrupted offload; wrong filesystem/table config (fileSystemProfile/uri) pointing at the wrong store; permissions on the offload directory.","solutions":["Verify the offload location (fileSystemUri/profile config) still points to the directory holding the offloaded ledger's index file","Check the index file exists and is intact for the reported ledgerId; re-offload the segment from the source ledger or restore from backup","Fix filesystem permissions for the broker user running tiered-storage reads","Re-run offload for the affected segment if the stored LedgerMetadata is corrupt/unparseable"],"exampleFix":"// config pointing at wrong store\noffloader: fileSystemUri: hdfs://old-cluster:9000\n// after\noffloader: fileSystemUri: hdfs://new-cluster:9000  (path restored/verified)","handlingStrategy":"try-catch","validationCode":"// verify the offload index exists before reading\nPath idx = Paths.get(offloadBaseDir, \"index\", ledgerId + \".index\");\nif (!Files.isRegularFile(idx) || Files.size(idx) == 0) {\n  throw new IllegalStateException(\"offload index missing for ledger \" + ledgerId);\n}","typeGuard":null,"tryCatchPattern":"try {\n  ReadHandle rh = FileStoreBackedReadHandleImpl.get(...);\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Fail to read LedgerMetadata\")) {\n    log.error(\"Offload index corrupt/missing for ledger — restore from backup or re-offload\");\n  } else throw e;\n}","preventionTips":["Monitor offload completion; alert on partially-written index files","Keep offload location config (fileSystemUri/profile) consistent across brokers","Back up offloaded index files; test restores","Verify filesystem permissions for the broker service user"],"tags":["io","tiered-storage","bookkeeper","pulsar"],"backgroundTag":"ledger-metadata-read-failure","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}