{"record":{"id":"b6142c07d777f439","repo":"apache/pulsar","slug":"failed-to-initialize-bookkeeper-metadata","errorCode":null,"errorMessage":"Failed to initialize BookKeeper metadata","messagePattern":"Failed to initialize BookKeeper metadata","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java","lineNumber":321,"sourceCode":"                arguments.metadataStoreConfigPath,\n                arguments.zkSessionTimeoutMillis);\n        MetadataStoreExtended configStore = initConfigMetadataStore(arguments.configurationMetadataStore,\n                arguments.configurationStoreConfigPath,\n                arguments.zkSessionTimeoutMillis);\n        try {\n\n        final String metadataStoreUrlNoIdentifier = MetadataStoreFactoryImpl\n                .removeIdentifierFromMetadataURL(arguments.metadataStoreUrl);\n        // Format BookKeeper ledger storage metadata\n        if (arguments.existingBkMetadataServiceUri == null && arguments.bookieMetadataServiceUri == null) {\n            ServerConfiguration bkConf = new ServerConfiguration();\n            bkConf.setListDelimiterHandler(new DisabledListDelimiterHandler());\n            bkConf.setMetadataServiceUri(\"metadata-store:\" + arguments.metadataStoreUrl);\n            bkConf.setZkTimeout(arguments.zkSessionTimeoutMillis);\n            // only format if /ledgers doesn't exist\n            if (!localStore.exists(BookKeeperConstants.DEFAULT_ZK_LEDGERS_ROOT_PATH).get()\n                && !BookKeeperAdmin.format(bkConf, false /* interactive */, false /* force */)) {\n                throw new IOException(\"Failed to initialize BookKeeper metadata\");\n            }\n        }\n\n        if (localStore instanceof DualMetadataStore && configStore instanceof DualMetadataStore) {\n            String uriStr;\n            if (arguments.existingBkMetadataServiceUri != null) {\n                uriStr = arguments.existingBkMetadataServiceUri;\n            } else if (arguments.bookieMetadataServiceUri != null) {\n                uriStr = arguments.bookieMetadataServiceUri;\n            } else {\n                uriStr = \"zk+null://\" + metadataStoreUrlNoIdentifier + BookKeeperConstants.DEFAULT_ZK_LEDGERS_ROOT_PATH;\n            }\n\n            // initial distributed log metadata\n            initialDlogNamespaceMetadata(arguments.configurationMetadataStore, uriStr);\n\n            ServiceURI bkMetadataServiceUri = ServiceURI.create(uriStr);\n            // Format BookKeeper stream storage metadata","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java#L303-L339","documentation":"PulsarClusterMetadataSetup.initializeCluster formats the BookKeeper metadata store (creating the /ledgers znode hierarchy) only if BookKeeperConstants.DEFAULT_ZK_LEDGERS_ROOT_PATH does not exist. If the path is absent and BookKeeperAdmin.format(bkConf, interactive=false, force=false) fails, this IOException is thrown, aborting cluster metadata initialization.","triggerScenarios":"Running 'pulsar initialize-cluster-metadata' against a metadata store where /ledgers doesn't exist and format fails — e.g. metadata service unreachable or refusing the format, ZK ACL/auth issues, or metadata-store URL misconfigured.","commonSituations":"ZooKeeper not running or wrong zkServers in the metadata store URL; partial previous initialization leaving inconsistent state; formatting while the BK metadata layout already exists but /ledgers is missing; connectivity/auth failures to the metadata store.","solutions":["Verify the metadata store (ZooKeeper) is reachable at the configured --zk-servers / metadata store URL","Inspect ZooKeeper for a partial layout (/admin and /ledgers); clean up stale state or use format with force via BookKeeperAdmin when safe","Re-run pulsar initialize-cluster-metadata after fixing connectivity","Check ZooKeeper ACLs/credentials allow creating the /ledgers node"],"exampleFix":"// before\nbkConf.setMetadataServiceUri(\"metadata-store:zk:wrong-host:2181\");\n// after\nbkConf.setMetadataServiceUri(\"metadata-store:zk:zk1:2181,zk2:2181,zk3:2181\");","handlingStrategy":"retry","validationCode":"MetadataStore store = PulsarMetadataStore.create(metadataStoreUrl);\nif (!store.exists(\"/ledgers\").get()) {\n    // verify connectivity before attempting format\n    store.getChildren(\"/\").get();\n}","typeGuard":null,"tryCatchPattern":"try {\n    initializeCluster(arguments, configStore, localStore);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"Failed to initialize BookKeeper metadata\")) {\n        throw new IllegalStateException(\"Check metadata store reachability and existing BK layout under /ledgers, then retry\", e);\n    }\n    throw e;\n}","preventionTips":["Confirm ZooKeeper/metadata store is up and reachable before running initialize-cluster-metadata","Inspect existing /admin and /ledgers znodes to detect partial prior initialization","Run cluster initialization as a deliberate, monitored step with retries on transient connectivity"],"tags":["bookkeeper","zookeeper","metadata","cluster-init"],"backgroundTag":"metadata-store-unavailable","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}