{"record":{"id":"9ad25ec60f0a9b0a","repo":"apache/pulsar","slug":"zookeeper-session-expired-give-up-to-become-audit","errorCode":null,"errorMessage":"Zookeeper session expired, give up to become auditor.","messagePattern":"Zookeeper session expired, give up to become auditor\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerAuditorManager.java","lineNumber":85,"sourceCode":"            notifyAll();\n        }\n    }\n\n    @Override\n    public void tryToBecomeAuditor(String bookieId, Consumer<AuditorEvent> listener) {\n        this.bookieId = bookieId;\n\n        LeaderElectionState les = leaderElection.elect(bookieId).join();\n\n        synchronized (this) {\n            leaderElectionState = les;\n        }\n\n        while (true) {\n            try {\n                synchronized (this) {\n                    if (sessionExpired) {\n                        throw new IllegalStateException(\"Zookeeper session expired, give up to become auditor.\");\n                    }\n                    if (leaderElectionState == LeaderElectionState.Leading) {\n                        return;\n                    } else {\n                        wait();\n                    }\n                }\n            } catch (InterruptedException e) {\n                throw new RuntimeException(e);\n            }\n        }\n    }\n\n    @Override\n    public BookieId getCurrentAuditor() {\n        return leaderElection.getLeaderValue()\n                .join()\n                .map(BookieId::parse)","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerAuditorManager.java#L67-L103","documentation":"tryToBecomeAuditor() waits in a loop to acquire the Pulsar auditor leader role. If the ZooKeeper session expires while the thread is waiting, the coordinator sets sessionExpired=true and the method throws IllegalStateException to signal that leadership acquisition has been abandoned and must be restarted with a fresh session.","triggerScenarios":"A thread calls tryToBecomeAuditor() and blocks in wait(); the ZooKeeper session is then expired (session loss, network partition, long GC pause exceeding session timeout), triggering the IllegalStateException on wake-up.","commonSituations":"Broker losing ZooKeeper connectivity during network flaps; ZooKeeper session timeout too short for JVM GC pauses; broker failover storms where many brokers contend for auditor and sessions lapse.","solutions":["Verify ZooKeeper session stability: increase sessionTimeoutMs if GC pauses or network hiccups exceed it.","Restart the auditor-election process after the broker re-establishes its ZooKeeper session (the coordinator typically re-runs tryToBecomeAuditor).","Check broker logs for SessionExpired/Disconnect events to correlate with GC pauses or network issues.","Ensure the broker's ZooKeeper client reconnect logic (session watcher) is healthy before re-attempting leadership."],"exampleFix":"// before\ncoordinator.tryToBecomeAuditor(); // throws IllegalStateException on session expiry\n// after\ntry {\n    coordinator.tryToBecomeAuditor();\n} catch (IllegalStateException e) {\n    LOG.warn(\"Auditor election aborted due to ZooKeeper session expiry; will retry after reconnect\", e);\n    // wait for session re-establishment, then re-attempt\n}","handlingStrategy":"try-catch","validationCode":"// Check session liveness before attempting election\nif (zkClient.getSessionId() == 0 || !zkClient.getState().isAlive()) {\n    throw new IllegalStateException(\"ZooKeeper session not alive; postpone auditor election\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    coordinator.tryToBecomeAuditor();\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"session expired\")) {\n        // wait for zk reconnect, then re-run election\n    } else {\n        throw e;\n    }\n}","preventionTips":["Tune sessionTimeoutMs above worst-case GC pause duration","Monitor ZooKeeper session expiry metrics on brokers","Re-attempt election only after the session watcher signals reconnected","Avoid network partitions between broker and ZK ensemble (colocate or harden links)"],"tags":["zookeeper","session-expired","leader-election","state"],"backgroundTag":"zookeeper-session-expired","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}