{"record":{"id":"bf3896c19371946c","repo":"apache/pulsar","slug":"exception-while-stopping-auto-ledger-re-replicatio","errorCode":null,"errorMessage":"Exception while stopping auto ledger re-replication","messagePattern":"Exception while stopping auto ledger re-replication","errorType":"exception","errorClass":"ReplicationException.UnavailableException","httpStatus":null,"severity":"error","filePath":"pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java","lineNumber":722,"sourceCode":"        } catch (TimeoutException ex) {\n            throw new ReplicationException.UnavailableException(\"Error contacting metadata store\", ex);\n        } catch (InterruptedException ie) {\n            Thread.currentThread().interrupt();\n            throw new ReplicationException.UnavailableException(\"Interrupted while connecting metadata store\", ie);\n        }\n    }\n\n    @Override\n    public void disableLedgerReplication()\n            throws ReplicationException.UnavailableException {\n        log.debug(\"disableLedgerReplication()\");\n        try {\n            store.put(replicationDisablePath, \"\".getBytes(UTF_8), Optional.of(-1L))\n                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);\n            log.info(\"Auto ledger re-replication is disabled!\");\n        } catch (ExecutionException | TimeoutException ee) {\n            log.error().exception(ee).log(\"Exception while stopping auto ledger re-replication\");\n            throw new ReplicationException.UnavailableException(\n                    \"Exception while stopping auto ledger re-replication\", ee);\n        } catch (InterruptedException ie) {\n            Thread.currentThread().interrupt();\n            throw new ReplicationException.UnavailableException(\n                    \"Interrupted while stopping auto ledger re-replication\", ie);\n        }\n    }\n\n    @Override\n    public void enableLedgerReplication()\n            throws ReplicationException.UnavailableException {\n        log.debug(\"enableLedgerReplication()\");\n        try {\n            store.delete(replicationDisablePath, Optional.empty())\n                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);\n            log.info(\"Resuming automatic ledger re-replication\");\n        } catch (ExecutionException | TimeoutException ee) {\n            log.error().exception(ee).log(\"Exception while resuming ledger replication\");","sourceCodeStart":704,"sourceCodeEnd":740,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java#L704-L740","documentation":"Thrown by disableLedgerReplication() when writing the replication-disable marker (store.put on replicationDisablePath) fails with ExecutionException or TimeoutException. BookKeeper cannot stop auto re-replication because the marker could not be persisted; wrapped as ReplicationException.UnavailableException.","triggerScenarios":"Calling disableLedgerReplication() (e.g. via autorecovery admin or bookie health check) while the metadata store is down, the put is rejected (no auth, read-only server, connection loss), or the future times out.","commonSituations":"ZooKeeper ensemble outage during a maintenance window where an operator disables re-replication, insufficient ACLs on the replication disable node, or client connected to a read-only ZooKeeper observer.","solutions":["Verify metadata store availability and ACL write permission on the replication disable path, then retry the disable.","Check the cause: auth/noauth or readonly errors require fixing credentials/client config; timeouts require network/quorum fixes.","Retry disableLedgerReplication() after recovery — the put is idempotent.","If during planned maintenance, ensure the metadata store is healthy before starting the procedure."],"exampleFix":"// before\nmanager.disableLedgerReplication();\n// after\ntry {\n    manager.disableLedgerReplication();\n} catch (ReplicationException.UnavailableException e) {\n    log.warn(\"Could not persist replication-disable marker; retrying\", e);\n    retryWithBackoff(() -> manager.disableLedgerReplication());\n}","handlingStrategy":"retry","validationCode":"// Java: pre-check store writability before disabling replication\nstore.exists(replicationDisablePath).get(5, TimeUnit.SECONDS);\n// plus: confirm ACLs allow put and server is not read-only","typeGuard":"static boolean storeFailure(ReplicationException.UnavailableException e) {\n    Throwable c = e.getCause();\n    return c instanceof ExecutionException || c instanceof TimeoutException;\n}","tryCatchPattern":"try {\n    manager.disableLedgerReplication();\n} catch (ReplicationException.UnavailableException e) {\n    backoffAndRetry(() -> manager.disableLedgerReplication()); // put is idempotent\n}","preventionTips":["Grant write ACLs on the replication-disable path to the bookie identity.","Confirm store health before maintenance windows that require disabling re-replication.","Retry idempotent puts with bounded backoff.","Alert on read-only/observer misconfiguration of the metadata store client."],"tags":["bookkeeper","metadata-store","autorecovery","disable"],"backgroundTag":"metadata-store-unavailable","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}