apache/pulsar · error · MetadataStoreException

Timed out waiting for migration participants to prepare: ${p

Error message

Timed out waiting for migration participants to prepare: ${pending}

What it means

During a migration, participants (broker/bookie components) must each mark themselves prepared before the coordinator proceeds. waitForPreparation polls with backoff until a deadline; if pending participants remain when System.currentTimeMillis() passes the deadline, it throws MetadataStoreException listing the still-pending set.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/MigrationCoordinator.java:173

    private void waitForPreparation() throws Exception {
        log.info("Waiting for all participants to prepare...");

        long deadline = System.currentTimeMillis() + preparationTimeout.toMillis();
        Backoff backoff = Backoff.builder()
                .initialDelay(Duration.ofMillis(100))
                .mandatoryStop(Duration.ofSeconds(60))
                .maxBackoff(Duration.ofSeconds(60)).build();
        while (true) {
            List<String> pending = sourceStore.getChildren(MigrationState.PARTICIPANTS_PATH).get();
            if (pending.isEmpty()) {
                log.info("All migration participants ready");
                return;
            }

            if (System.currentTimeMillis() >= deadline) {
                log.error().attr("pendingParticipants", pending)
                        .log("Failed to wait for all participants to prepare");
                throw new MetadataStoreException(
                        "Timed out waiting for migration participants to prepare: " + pending);
            }

            log.info().attr("pending", pending).log("Waiting for participants to prepare");
            Thread.sleep(backoff.next().toMillis());
        }
    }

    private void copyPersistentData() throws Exception {
        log.info("Starting persistent data copy...");

        AtomicLong copiedCount = new AtomicLong(0);
        Semaphore semaphore = new Semaphore(MAX_PENDING_OPS);
        AtomicReference<Throwable> exception = new AtomicReference<>();

        // Bootstrap first level
        BlockingQueue<String> workQueue = new LinkedBlockingQueue<>(getChildren("/").get());

View on GitHub (pinned to 820761864e)

Solutions

1. Inspect the `pending` list in the error/log to identify which participants did not prepare
2. Start/repair the listed brokers/bookies and ensure they run a Pulsar version supporting the migration protocol, then re-run startMigration
3. Increase preparationTimeout if the cluster legitimately needs more time
4. Verify participant connectivity to the source metadata store during the migration

Example fix

// before
new MigrationCoordinator(source, targetUrl, Duration.ofSeconds(60)); // 1 broker is restarting
// after
new MigrationCoordinator(source, targetUrl, Duration.ofMinutes(10));

// Java: verify all participants are up and migration-capable before starting
for (String participant : expectedParticipants) {
    if (!isReachable(participant)) {
        throw new IllegalStateException("Participant not ready: " + participant);
    }
}

try {
    coordinator.startMigration();
} catch (MetadataStoreException e) {
    if (e.getMessage().startsWith("Timed out waiting for migration participants")) {
        // restart/repair pending participants, then re-run; operation is restartable
    }
}

When it happens

Trigger: One or more migration participants (brokers/bookies) fail to write their prepared flag within preparationTimeout — because they are down, disconnected from the source store, running an old version without migration support, or simply slower than the timeout.

Common situations: A broker is stopped during the migration window; cluster nodes on pre-PIP-462 versions that never prepare; slow network or overloaded metadata store delaying participant updates past the deadline.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• RestException(e)
• Time-out while checking authorization
• Failed to validate global cluster configuration : ns=%s ems
• Failed to get children of ${path}
• Failed to get data from ${path}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/18076465089e9094. Report an issue: GitHub.