{"record":{"id":"dd806fe1792be426","repo":"apache/pulsar","slug":"migration-is-already-in-progress-phase-phase","errorCode":null,"errorMessage":"Migration is already in progress (phase: ${phase})","messagePattern":"Migration is already in progress \\(phase: (.+?)\\)","errorType":"http","errorClass":"RestException","httpStatus":409,"severity":"error","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/MetadataMigrationBase.java","lineNumber":110,"sourceCode":"            throw new RestException(Response.Status.BAD_REQUEST, \"Target URL is required\");\n        }\n\n        try {\n            // Check if metadata store is wrapped with DualMetadataStore\n            if (!(pulsar().getLocalMetadataStore() instanceof DualMetadataStore dualStore)) {\n                throw new RestException(Response.Status.BAD_REQUEST, \"Metadata store is not configured for migration. \"\n                        + \"Please ensure you're using a supported source metadata store (e.g., ZooKeeper).\");\n            }\n\n            // Reject the request if a migration is already in progress or was completed. The migration\n            // flag is always kept in the source store, so read it from there: after a completed\n            // migration the dual store would route the read to the target store.\n            var existingFlag = dualStore.getSourceStore().get(MigrationState.MIGRATION_FLAG_PATH).get();\n            if (existingFlag.isPresent()) {\n                MigrationState currentState = ObjectMapperFactory.getMapper().reader()\n                        .readValue(existingFlag.get().getValue(), MigrationState.class);\n                switch (currentState.getPhase()) {\n                    case PREPARATION, COPYING -> throw new RestException(Response.Status.CONFLICT,\n                            \"Migration is already in progress (phase: \" + currentState.getPhase() + \")\");\n                    case COMPLETED -> throw new RestException(Response.Status.CONFLICT,\n                            \"Migration has already been completed\");\n                    default -> {\n                        // NOT_STARTED or FAILED: ok to start (or retry) the migration\n                    }\n                }\n            }\n\n            // Create coordinator\n            MigrationCoordinator coordinator = new MigrationCoordinator(pulsar().getLocalMetadataStore(), targetUrl);\n\n            // Start migration in background thread\n            pulsar().getExecutor().submit(() -> {\n                try {\n                    log.info().attr(\"targetUrl\", targetUrl).log(\"Starting metadata migration\");\n                    coordinator.startMigration();\n                    log.info(\"Metadata migration completed successfully\");","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/MetadataMigrationBase.java#L92-L128","documentation":"A 409 CONFLICT thrown when a metadata migration is started while one is already running in phase PREPARATION or COPYING. The migration state flag stored in the source metadata store records the current phase, and concurrent migrations are rejected to avoid two migrations interleaving.","triggerScenarios":"Calling the migration start endpoint twice while the first run is still in PREPARATION or COPYING; a previous run hung/stuck leaving the flag in PREPARATION/COPYING; a client retrying with aggressive backoff.","commonSituations":"Automation scripts without idempotency triggering duplicate starts; a crashed migration leaving the flag stuck mid-phase so subsequent starts always conflict; multiple operators starting migrations simultaneously.","solutions":["Wait for the running migration to reach COMPLETED or FAILED, polling the migration status endpoint.","If the migration is genuinely stuck, cancel/reset the migration via the provided cancel/abort endpoint so the flag returns to a restartable state (FAILED/NOT_STARTED).","Inspect the stored MigrationState at MigrationState.MIGRATION_FLAG_PATH to confirm the real phase before restarting brokers/stores.","As a last resort, carefully delete the migration flag from the source store only after confirming no migration is running (data-loss risk: do this only with the migration halted)."],"exampleFix":"// before: blind retry\nstartMigration(target); // 409\n// after: check status first\nMigrationState s = getMigrationStatus();\nif (s.getPhase() == Phase.NOT_STARTED || s.getPhase() == Phase.FAILED) {\n    startMigration(target);\n}","handlingStrategy":"retry","validationCode":"MigrationState s = getMigrationStatus();\nif (s.getPhase() == Phase.PREPARATION || s.getPhase() == Phase.COPYING) {\n  throw new IllegalStateException(\"migration already running; wait or cancel first\");\n}","typeGuard":null,"tryCatchPattern":"try { startMigration(target); }\ncatch (PulsarAdminException e) {\n  if (e.getStatusCode() == 409) {\n    // poll status until COMPLETED/FAILED, then decide whether to restart\n  } else throw e;\n}","preventionTips":["Serialize migration start calls (single operator/runbook owner)","Check migration status before starting","Add idempotent guards in automation scripts","Investigate stuck PREPARATION/COPYING states promptly via the flag at MigrationState.MIGRATION_FLAG_PATH"],"tags":["metadata-migration","conflict","state"],"backgroundTag":"migration-already-in-progress","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}