{"record":{"id":"7359e7ea2e4a1d0b","repo":"apache/pulsar","slug":"failed-to-get-cluster-instance-id","errorCode":null,"errorMessage":"Failed to get cluster instance id","messagePattern":"Failed to get cluster instance id","errorType":"exception","errorClass":"BookieException.MetadataStoreException","httpStatus":null,"severity":"error","filePath":"pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarRegistrationManager.java","lineNumber":128,"sourceCode":"            }\n        }\n        try {\n            coordinationService.close();\n        } catch (Exception e) {\n            throw new RuntimeException(e);\n        }\n    }\n\n    @Override\n    public String getClusterInstanceId() throws BookieException {\n        try {\n            return store.get(ledgersRootPath + \"/\" + INSTANCEID)\n                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS)\n                    .map(res -> new String(res.getValue(), UTF_8))\n                    .orElseThrow(\n                            () -> new BookieException.MetadataStoreException(\"BookKeeper cluster not initialized\"));\n        } catch (ExecutionException | InterruptedException | TimeoutException e) {\n            throw new BookieException.MetadataStoreException(\"Failed to get cluster instance id\", e);\n        }\n    }\n\n    @Override\n    public void registerBookie(BookieId bookieId, boolean readOnly, BookieServiceInfo bookieServiceInfo)\n            throws BookieException {\n        String regPath = bookieRegistrationPath + \"/\" + bookieId;\n        String regPathReadOnly = bookieReadonlyRegistrationPath + \"/\" + bookieId;\n        log.info().attr(\"bookieId\", bookieId).attr(\"readOnly\", readOnly).attr(\"info\", bookieServiceInfo)\n                .log(\"RegisterBookie\");\n\n        try {\n            if (readOnly) {\n                ResourceLock<BookieServiceInfo> rwRegistration = bookieRegistration.remove(bookieId);\n                if (rwRegistration != null) {\n                    log.info().attr(\"bookieId\", bookieId)\n                            .log(\"Bookie was already registered as writable, unregistering\");\n                    rwRegistration.release().get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarRegistrationManager.java#L110-L146","documentation":"Thrown by PulsarRegistrationManager.getClusterInstanceId when reading the instanceid znode under the ledgers root fails — the async store.get either errored (ExecutionException), was interrupted, or exceeded BLOCKING_CALL_TIMEOUT. It is wrapped in BookieException.MetadataStoreException. BookKeeper uses this value during bookie registration to verify all bookies belong to the same cluster, so registration cannot proceed without it.","triggerScenarios":"Calling getClusterInstanceId() (during bookie startup/registration) when the metadata store is unreachable, the read times out, or the calling thread is interrupted while waiting on the INSTANCEID znode read.","commonSituations":"Bookie starting before ZooKeeper/metadata service is available; wrong metadataServiceUrl in bookie.conf; ZooKeeper quorum loss or session expiry; network firewall/DNS issues on the bookie host; cluster not fully initialized so instanceid is missing entirely (a distinct sibling error, 'BookKeeper cluster not initialized').","solutions":["Verify the metadata store is healthy and reachable from the bookie (zkCli or pulsar metadata tool) and fix connectivity/quorum issues.","Check metadataServiceUrl in bookie.conf points at the correct ZooKeeper/metadata service and ledgers root.","Restart the bookie after the metadata store recovers — registration is retried on bookie restart.","If timeouts are frequent, check ensemble load and latency; ensure the INSTANCEID znode exists (cluster initialized via bin/bookkeeper shell init)."],"exampleFix":"// before\n// bookie.conf: metadataServiceUrl=zk+host:2181/ledgers  (wrong host)\n// after\n// bookie.conf: metadataServiceUrl=zk+zk1:2181,zk2:2181,zk3:2181/ledgers\n// plus readiness gate before starting the bookie\n// while !nc -z zk1 2181; do sleep 2; done; bin/pulsar bookie start","handlingStrategy":"validation","validationCode":"// before bookie startup: verify store reachable and cluster initialized\nMetadataStore store = PulsarMetadataStore.instance(metadataServiceUrl);\nboolean ready = store.get(ledgersRootPath + \"/instanceid\")\n        .get(30, TimeUnit.SECONDS).isPresent();\nif (!ready) throw new IllegalStateException(\"Cluster not initialized: run 'bin/bookkeeper shell initbookie' first\");","typeGuard":null,"tryCatchPattern":"try {\n    return registrationManager.getClusterInstanceId();\n} catch (BookieException.MetadataStoreException e) {\n    if (e.getCause() instanceof TimeoutException || e.getCause() instanceof ExecutionException) {\n        return retryWithBackoff(registrationManager::getClusterInstanceId, 3); // transient store outage\n    }\n    throw e;\n}","preventionTips":["Add a readiness check (ZooKeeper port/quorum) before starting the bookie process.","Double-check metadataServiceUrl in bookie.conf, including the ledgers root path.","Initialize the cluster once (bookkeeper shell initbookie) so the instanceid znode exists.","Run bookies on hosts with stable DNS/network paths to the metadata service and monitor session expiry."],"tags":["zookeeper","bookie-registration","metadata-store","timeout"],"backgroundTag":"zookeeper-unreachable","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}