apache/pulsar · error · RestException

Migration is already in progress (phase: ${phase})

Error message

Migration is already in progress (phase: ${phase})

What it means

A 409 CONFLICT thrown when a metadata migration is started while one is already running in phase PREPARATION or COPYING. The migration state flag stored in the source metadata store records the current phase, and concurrent migrations are rejected to avoid two migrations interleaving.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/MetadataMigrationBase.java:110

            throw new RestException(Response.Status.BAD_REQUEST, "Target URL is required");
        }

        try {
            // Check if metadata store is wrapped with DualMetadataStore
            if (!(pulsar().getLocalMetadataStore() instanceof DualMetadataStore dualStore)) {
                throw new RestException(Response.Status.BAD_REQUEST, "Metadata store is not configured for migration. "
                        + "Please ensure you're using a supported source metadata store (e.g., ZooKeeper).");
            }

            // Reject the request if a migration is already in progress or was completed. The migration
            // flag is always kept in the source store, so read it from there: after a completed
            // migration the dual store would route the read to the target store.
            var existingFlag = dualStore.getSourceStore().get(MigrationState.MIGRATION_FLAG_PATH).get();
            if (existingFlag.isPresent()) {
                MigrationState currentState = ObjectMapperFactory.getMapper().reader()
                        .readValue(existingFlag.get().getValue(), MigrationState.class);
                switch (currentState.getPhase()) {
                    case PREPARATION, COPYING -> throw new RestException(Response.Status.CONFLICT,
                            "Migration is already in progress (phase: " + currentState.getPhase() + ")");
                    case COMPLETED -> throw new RestException(Response.Status.CONFLICT,
                            "Migration has already been completed");
                    default -> {
                        // NOT_STARTED or FAILED: ok to start (or retry) the migration
                    }
                }
            }

            // Create coordinator
            MigrationCoordinator coordinator = new MigrationCoordinator(pulsar().getLocalMetadataStore(), targetUrl);

            // Start migration in background thread
            pulsar().getExecutor().submit(() -> {
                try {
                    log.info().attr("targetUrl", targetUrl).log("Starting metadata migration");
                    coordinator.startMigration();
                    log.info("Metadata migration completed successfully");

View on GitHub (pinned to 820761864e)

Solutions

1. Wait for the running migration to reach COMPLETED or FAILED, polling the migration status endpoint.
2. If the migration is genuinely stuck, cancel/reset the migration via the provided cancel/abort endpoint so the flag returns to a restartable state (FAILED/NOT_STARTED).
3. Inspect the stored MigrationState at MigrationState.MIGRATION_FLAG_PATH to confirm the real phase before restarting brokers/stores.
4. As a last resort, carefully delete the migration flag from the source store only after confirming no migration is running (data-loss risk: do this only with the migration halted).

Example fix

// before: blind retry
startMigration(target); // 409
// after: check status first
MigrationState s = getMigrationStatus();
if (s.getPhase() == Phase.NOT_STARTED || s.getPhase() == Phase.FAILED) {
    startMigration(target);
}

MigrationState s = getMigrationStatus();
if (s.getPhase() == Phase.PREPARATION || s.getPhase() == Phase.COPYING) {
  throw new IllegalStateException("migration already running; wait or cancel first");
}

try { startMigration(target); }
catch (PulsarAdminException e) {
  if (e.getStatusCode() == 409) {
    // poll status until COMPLETED/FAILED, then decide whether to restart
  } else throw e;
}

When it happens

Trigger: Calling the migration start endpoint twice while the first run is still in PREPARATION or COPYING; a previous run hung/stuck leaving the flag in PREPARATION/COPYING; a client retrying with aggressive backoff.

Common situations: Automation scripts without idempotency triggering duplicate starts; a crashed migration leaving the flag stuck mid-phase so subsequent starts always conflict; multiple operators starting migrations simultaneously.

Related errors

• Migration has already been completed
• ManagedLedgerFactory is already closed.
• ManagedLedger ${name} has already been closed
• Cluster already exists
• ${duplicateBrokers} already exists in ${domainName}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/dd806fe1792be426. Report an issue: GitHub.