apache/pulsar · error · RestException

Worker ${workerId} was not yet removed after a prior drain o

Error message

Worker ${workerId} was not yet removed after a prior drain op; try later

What it means

This HTTP 412 (Precondition Failed) error is returned by the drain endpoint when the target worker had a prior drain operation but has not yet been removed from cluster membership. SchedulerManager requires the previously-drained worker to be fully deregistered before another drain can proceed.

Source

Thrown at pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java:279

                .log("drain called");

        throwIfNotSuperUser(authParams, "drain worker");

        // Depending on which operations we decide to allow, we may add checks here to error/exception if
        //      calledOnLeaderUri is true on a non-leader
        //      calledOnLeaderUri is false on a leader
        // For now, deal with everything.

        if (worker().getLeaderService().isLeader()) {
            try {
                worker().getSchedulerManager().drainIfNotInProgress(workerId);
            } catch (SchedulerManager.DrainInProgressException e) {
                throw new RestException(Status.CONFLICT, "Another drain is in progress");
            } catch (SchedulerManager.TooFewWorkersException e) {
                throw new RestException(Status.BAD_REQUEST, "Too few workers (need at least 2)");
            } catch (SchedulerManager.WorkerNotRemovedAfterPriorDrainException e) {
                String errString = "Worker " + workerId + " was not yet removed after a prior drain op; try later";
                throw new RestException(Status.PRECONDITION_FAILED, errString);
            } catch (SchedulerManager.UnknownWorkerException e) {
                String errString = "Worker " + workerId + " is not among the current workers in the system";
                throw new RestException(Status.BAD_REQUEST, errString);
            }
        } else {
            URI redirect = buildRedirectUriForDrainRelatedOp(uri, workerId);
            log.info().attr("redirect", redirect).log("Not leader; redirect URI=");
            throw new WebApplicationException(Response.temporaryRedirect(redirect).build());
        }
    }

    @Override
    public LongRunningProcessStatus getDrainStatus(final URI uri, final String inWorkerId,
                                                   final AuthenticationParameters authParams,
                                                   boolean calledOnLeaderUri) {
        if (!isWorkerServiceAvailable()) {
            throwUnavailableException();
        }

View on GitHub (pinned to 820761864e)

Solutions

1. Wait for the worker to be removed from membership (check GET /admin/v2/worker/cluster) and retry later
2. Restart/stop the drained worker process so it deregisters cleanly
3. Investigate the worker's shutdown path if it stays registered indefinitely

boolean stillRegistered = admin.functions().getCluster().stream()
    .anyMatch(w -> w.getWorkerId().equals(workerId)); // re-drain only after removal

try {
    admin.functions().drain(workerId);
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 412) {
        // poll membership until the drained worker is removed, then retry
    }
}

When it happens

Trigger: Calling PUT /admin/v2/worker/drain/{workerId} for a workerId whose earlier drain completed but whose WorkerInfo is still present in the membership manager (e.g. the worker process hasn't fully deregistered or its membership entry hasn't expired).

Common situations: Re-draining the same worker shortly after a previous drain; a hung worker process that drained its functions but never shut down cleanly; stale membership entries in the metadata store.

Related errors

• Another drain is in progress
• Worker ${workerId} is not among the current workers in the s
• cluster data is required
• Cluster already exists
• Peer cluster ${peerCluster} does not exist

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/b0b762bfae6e9d4b. Report an issue: GitHub.