apache/pulsar · error · MetadataStoreException

Migration is already in progress (phase: ${phase})

Error message

Migration is already in progress (phase: ${phase})

What it means

startMigration() first writes a MigrationState flag at MIGRATION_FLAG_PATH on the source store. If a flag already exists with phase PREPARATION or COPYING, another migration is mid-flight, so setInitialMigrationPhase throws MetadataStoreException instead of clobbering the in-progress state. The expected-version guard exists precisely to prevent concurrent starts.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/MigrationCoordinator.java:129

            throw e;
        }
    }

    private void setInitialMigrationPhase() throws MetadataStoreException {
        try {
            Optional<GetResult> existing = sourceStore.get(MigrationState.MIGRATION_FLAG_PATH).get();
            Optional<Long> expectedVersion;
            if (existing.isEmpty()) {
                // Create-only, to guard against concurrent migration starts
                expectedVersion = Optional.of(-1L);
            } else {
                MigrationState currentState = ObjectMapperFactory.getMapper().reader()
                        .readValue(existing.get().getValue(), MigrationState.class);
                expectedVersion = switch (currentState.getPhase()) {
                    // A leftover flag from a failed (or never started) migration can be replaced. The
                    // expected version guards against concurrent migration starts.
                    case NOT_STARTED, FAILED -> Optional.of(existing.get().getStat().getVersion());
                    case PREPARATION, COPYING -> throw new MetadataStoreException(
                            "Migration is already in progress (phase: " + currentState.getPhase() + ")");
                    case COMPLETED -> throw new MetadataStoreException("Migration has already been completed");
                };
            }

            sourceStore.put(MigrationState.MIGRATION_FLAG_PATH,
                    ObjectMapperFactory.getMapper().writer()
                            .writeValueAsBytes(new MigrationState(MigrationPhase.PREPARATION, targetUrl)),
                    expectedVersion).get();
        } catch (MetadataStoreException e) {
            throw e;
        } catch (Exception e) {
            throw new MetadataStoreException(e);
        }
    }

    private void updatePhase(MigrationPhase phase) throws MetadataStoreException {
        try {

View on GitHub (pinned to 820761864e)

Solutions

1. Check the MigrationState at MIGRATION_FLAG_PATH on the source store to see who/when set it
2. If the migration is genuinely stuck, manually reset the flag (delete or rewrite MIGRATION_FLAG_PATH with phase NOT_STARTED) and restart
3. Wait for the in-progress migration to reach COMPLETED or FAILED before starting a new one
4. Ensure only one operator/tool instance drives the migration at a time

Example fix

// before
coordinator.startMigration(); // throws: phase COPYING left over from crashed run
// after
coordinator.resetMigrationFlag(); // set MigrationState(NOT_STARTED)
coordinator.startMigration();

// Java
Optional<Versioned<byte[]>> flag =
    sourceStore.get(MigrationState.MIGRATION_FLAG_PATH).get(30, TimeUnit.SECONDS);
if (flag.isPresent()) {
    MigrationState s = ObjectMapperFactory.getMapper().reader()
        .readValue(flag.get().getValue(), MigrationState.class);
    if (s.getPhase() == MigrationPhase.PREPARATION || s.getPhase() == MigrationPhase.COPYING) {
        throw new IllegalStateException("Migration already running in phase " + s.getPhase());
    }
}

boolean migrationInProgress(MetadataStore src) throws Exception {
    Optional<Versioned<byte[]>> f = src.get(MigrationState.MIGRATION_FLAG_PATH).get(30, TimeUnit.SECONDS);
    return f.isPresent() && (readPhase(f.get()) == MigrationPhase.PREPARATION
        || readPhase(f.get()) == MigrationPhase.COPYING);
}

try {
    coordinator.startMigration();
} catch (MetadataStoreException e) {
    if (e.getMessage().contains("already in progress")) {
        // inspect MIGRATION_FLAG_PATH; reset flag only if the run is confirmed dead
    }
}

When it happens

Trigger: Calling startMigration while a previous startMigration is in PREPARATION or COPYING phase (still running, hung, or its process died without advancing/resetting the flag).

Common situations: Re-running a migration script after a crashed attempt while the flag was left in PREPARATION; two operators starting the migration simultaneously.

Related errors

• Expected target metadata store to be Oxia
• Migration has already been completed
• Concurrent modification
• Failed to set data for ${path}
• Failed to set/create ${path}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8ea27b0e6d5a3f8b. Report an issue: GitHub.