{"record":{"id":"66f685d7ebad0380","repo":"apache/pulsar","slug":"error-contacting-metadata-store","errorCode":null,"errorMessage":"Error contacting metadata store","messagePattern":"Error contacting metadata store","errorType":"exception","errorClass":"ReplicationException.UnavailableException","httpStatus":null,"severity":"error","filePath":"pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java","lineNumber":479,"sourceCode":"                                || (ee.getCause() instanceof MetadataStoreException\n                                        && ee.getCause().getCause()\n                                                instanceof KeeperException.NotEmptyException);\n                        if (!isNotEmpty) {\n                            log.warn().exception(ee).log(\"Error deleting underreplicated ledger parent node\");\n                        }\n                    }\n                }\n            }\n        } catch (ExecutionException ee) {\n            if (ee.getCause() instanceof MetadataStoreException.NotFoundException) {\n                // this is ok\n            } else if (ee.getCause() instanceof MetadataStoreException.BadVersionException) {\n                // if this is the case, some has marked the ledger\n                // for rereplication again. Leave the underreplicated\n                // znode in place, so the ledger is checked.\n            } else {\n                log.error().exception(ee).log(\"Error deleting underreplicated ledger node\");\n                throw new ReplicationException.UnavailableException(\"Error contacting metadata store\", ee);\n            }\n        } catch (TimeoutException ex) {\n            throw new ReplicationException.UnavailableException(\"Error contacting metadata store\", ex);\n        } catch (InterruptedException ie) {\n            Thread.currentThread().interrupt();\n            throw new ReplicationException.UnavailableException(\"Interrupted while contacting metadata store\", ie);\n        } finally {\n            releaseUnderreplicatedLedger(ledgerId);\n        }\n    }\n\n    /**\n     * Get a list of all the underreplicated ledgers which have been\n     * marked for rereplication, filtered by the predicate on the replicas list.\n     *\n     * <p>Replicas list of an underreplicated ledger is the list of the bookies which are part of\n     * the ensemble of this ledger and are currently unavailable/down.\n     *","sourceCodeStart":461,"sourceCodeEnd":497,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java#L461-L497","documentation":"Thrown by markLedgerReplicated when deleting the under-replicated ledger node (store.delete with expected version) fails with an ExecutionException whose cause is neither NotFoundException nor BadVersionException (both benign), or when the delete/parent-cleanup call times out. It means the metadata store could not confirm removal of the marker, so the ledger may remain flagged as under-replicated even though replication succeeded.","triggerScenarios":"Calling markLedgerReplicated(ledgerId) after successful rereplication when: the delete fails with an unexpected MetadataStoreException (connection lost, session expired, ACL/permission denied), or the blocking get/delete exceeds BLOCKING_CALL_TIMEOUT. NotFound (already deleted) and BadVersion (someone re-marked the ledger) are intentionally not errors.","commonSituations":"ZooKeeper session expiry mid-cleanup after a long replication run; permission/ACL changes on the /ledgers/underreplicated subtree; metadata store outage during auditor cleanup; the ledger being re-marked by a concurrent audit while being cleaned (benign cases pass through, others surface here).","solutions":["Retry markLedgerReplicated (or verify the node's state) once the metadata store is reachable; if NotFoundException is then returned, the marker is gone and the operation effectively succeeded.","Check ee.getCause() for MetadataStoreException type: session/auth issues require reconnection or ACL fixes; transient IO errors warrant retry.","Verify write permissions (ACLs) on the under-replicated ledger path if the cause indicates permission failure.","Confirm the ledger actually replicated (via the admin API) and rely on the Auditor to re-check the leftover marker if the node remains."],"exampleFix":"// before: single attempt, fails hard on transient store error\nurManager.markLedgerReplicated(ledgerId);\n// after: retry transient failures and confirm the marker state\ntry {\n    urManager.markLedgerReplicated(ledgerId);\n} catch (ReplicationException.UnavailableException e) {\n    if (isTransient(e.getCause())) {\n        retryWithBackoff(() -> urManager.markLedgerReplicated(ledgerId));\n    } else {\n        throw e;\n    }\n}","handlingStrategy":"retry","validationCode":"// verify the marker still exists and matches the held lock version before deleting\nvar res = metadataStore.get(urLedgerPath).join();\nif (res.isEmpty()) { return; /* already replicated — nothing to do */ }\nlong version = res.get().getStat().getVersion();","typeGuard":"static boolean isBenignCause(ExecutionException ee) {\n    return ee.getCause() instanceof MetadataStoreException.NotFoundException\n        || ee.getCause() instanceof MetadataStoreException.BadVersionException;\n}","tryCatchPattern":"try {\n    urManager.markLedgerReplicated(ledgerId);\n} catch (ReplicationException.UnavailableException e) {\n    Throwable c = e.getCause();\n    if (isTransient(c)) { retryWithBackoff(() -> urManager.markLedgerReplicated(ledgerId)); }\n    else { log.error(\"non-transient delete failure for {}\", ledgerId, e); }\n}","preventionTips":["Keep ZK sessions alive during long replication runs (avoid long GC pauses)","Maintain consistent ACLs on the /ledgers/underreplicated subtree","Reconcile leftover markers with a later audit pass instead of treating them as fatal","Check ee.getCause() type before retrying: NotFound/BadVersion need no retry"],"tags":["metadata-store","delete-failed","zookeeper","bookkeeper"],"backgroundTag":"metadata-store-unreachable","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}