{"record":{"id":"18076465089e9094","repo":"apache/pulsar","slug":"timed-out-waiting-for-migration-participants-to-pr","errorCode":null,"errorMessage":"Timed out waiting for migration participants to prepare: ${pending}","messagePattern":"Timed out waiting for migration participants to prepare: (.+?)","errorType":"exception","errorClass":"MetadataStoreException","httpStatus":null,"severity":"error","filePath":"pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/MigrationCoordinator.java","lineNumber":173,"sourceCode":"    private void waitForPreparation() throws Exception {\n        log.info(\"Waiting for all participants to prepare...\");\n\n        long deadline = System.currentTimeMillis() + preparationTimeout.toMillis();\n        Backoff backoff = Backoff.builder()\n                .initialDelay(Duration.ofMillis(100))\n                .mandatoryStop(Duration.ofSeconds(60))\n                .maxBackoff(Duration.ofSeconds(60)).build();\n        while (true) {\n            List<String> pending = sourceStore.getChildren(MigrationState.PARTICIPANTS_PATH).get();\n            if (pending.isEmpty()) {\n                log.info(\"All migration participants ready\");\n                return;\n            }\n\n            if (System.currentTimeMillis() >= deadline) {\n                log.error().attr(\"pendingParticipants\", pending)\n                        .log(\"Failed to wait for all participants to prepare\");\n                throw new MetadataStoreException(\n                        \"Timed out waiting for migration participants to prepare: \" + pending);\n            }\n\n            log.info().attr(\"pending\", pending).log(\"Waiting for participants to prepare\");\n            Thread.sleep(backoff.next().toMillis());\n        }\n    }\n\n    private void copyPersistentData() throws Exception {\n        log.info(\"Starting persistent data copy...\");\n\n        AtomicLong copiedCount = new AtomicLong(0);\n        Semaphore semaphore = new Semaphore(MAX_PENDING_OPS);\n        AtomicReference<Throwable> exception = new AtomicReference<>();\n\n        // Bootstrap first level\n        BlockingQueue<String> workQueue = new LinkedBlockingQueue<>(getChildren(\"/\").get());\n","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/MigrationCoordinator.java#L155-L191","documentation":"During a migration, participants (broker/bookie components) must each mark themselves prepared before the coordinator proceeds. waitForPreparation polls with backoff until a deadline; if pending participants remain when System.currentTimeMillis() passes the deadline, it throws MetadataStoreException listing the still-pending set.","triggerScenarios":"One or more migration participants (brokers/bookies) fail to write their prepared flag within preparationTimeout — because they are down, disconnected from the source store, running an old version without migration support, or simply slower than the timeout.","commonSituations":"A broker is stopped during the migration window; cluster nodes on pre-PIP-462 versions that never prepare; slow network or overloaded metadata store delaying participant updates past the deadline.","solutions":["Inspect the `pending` list in the error/log to identify which participants did not prepare","Start/repair the listed brokers/bookies and ensure they run a Pulsar version supporting the migration protocol, then re-run startMigration","Increase preparationTimeout if the cluster legitimately needs more time","Verify participant connectivity to the source metadata store during the migration"],"exampleFix":"// before\nnew MigrationCoordinator(source, targetUrl, Duration.ofSeconds(60)); // 1 broker is restarting\n// after\nnew MigrationCoordinator(source, targetUrl, Duration.ofMinutes(10));","handlingStrategy":"retry","validationCode":"// Java: verify all participants are up and migration-capable before starting\nfor (String participant : expectedParticipants) {\n    if (!isReachable(participant)) {\n        throw new IllegalStateException(\"Participant not ready: \" + participant);\n    }\n}\n","typeGuard":null,"tryCatchPattern":"try {\n    coordinator.startMigration();\n} catch (MetadataStoreException e) {\n    if (e.getMessage().startsWith(\"Timed out waiting for migration participants\")) {\n        // restart/repair pending participants, then re-run; operation is restartable\n    }\n}\n","preventionTips":["Ensure every broker/bookie runs a Pulsar version supporting the migration protocol","Increase preparationTimeout for large or slow clusters","Verify participant connectivity to the source metadata store before starting","Restart crashed participants before re-running startMigration"],"tags":["migration","timeout","metadata-store","distributed-coordination"],"backgroundTag":"migration-participant-preparation-timeout","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}