{"record":{"id":"b0b762bfae6e9d4b","repo":"apache/pulsar","slug":"worker-workerid-was-not-yet-removed-after-a-pri-b0b762","errorCode":null,"errorMessage":"Worker ${workerId} was not yet removed after a prior drain op; try later","messagePattern":"Worker (.+?) was not yet removed after a prior drain op; try later","errorType":"http","errorClass":"RestException","httpStatus":412,"severity":"error","filePath":"pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java","lineNumber":279,"sourceCode":"                .log(\"drain called\");\n\n        throwIfNotSuperUser(authParams, \"drain worker\");\n\n        // Depending on which operations we decide to allow, we may add checks here to error/exception if\n        //      calledOnLeaderUri is true on a non-leader\n        //      calledOnLeaderUri is false on a leader\n        // For now, deal with everything.\n\n        if (worker().getLeaderService().isLeader()) {\n            try {\n                worker().getSchedulerManager().drainIfNotInProgress(workerId);\n            } catch (SchedulerManager.DrainInProgressException e) {\n                throw new RestException(Status.CONFLICT, \"Another drain is in progress\");\n            } catch (SchedulerManager.TooFewWorkersException e) {\n                throw new RestException(Status.BAD_REQUEST, \"Too few workers (need at least 2)\");\n            } catch (SchedulerManager.WorkerNotRemovedAfterPriorDrainException e) {\n                String errString = \"Worker \" + workerId + \" was not yet removed after a prior drain op; try later\";\n                throw new RestException(Status.PRECONDITION_FAILED, errString);\n            } catch (SchedulerManager.UnknownWorkerException e) {\n                String errString = \"Worker \" + workerId + \" is not among the current workers in the system\";\n                throw new RestException(Status.BAD_REQUEST, errString);\n            }\n        } else {\n            URI redirect = buildRedirectUriForDrainRelatedOp(uri, workerId);\n            log.info().attr(\"redirect\", redirect).log(\"Not leader; redirect URI=\");\n            throw new WebApplicationException(Response.temporaryRedirect(redirect).build());\n        }\n    }\n\n    @Override\n    public LongRunningProcessStatus getDrainStatus(final URI uri, final String inWorkerId,\n                                                   final AuthenticationParameters authParams,\n                                                   boolean calledOnLeaderUri) {\n        if (!isWorkerServiceAvailable()) {\n            throwUnavailableException();\n        }","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java#L261-L297","documentation":"This HTTP 412 (Precondition Failed) error is returned by the drain endpoint when the target worker had a prior drain operation but has not yet been removed from cluster membership. SchedulerManager requires the previously-drained worker to be fully deregistered before another drain can proceed.","triggerScenarios":"Calling PUT /admin/v2/worker/drain/{workerId} for a workerId whose earlier drain completed but whose WorkerInfo is still present in the membership manager (e.g. the worker process hasn't fully deregistered or its membership entry hasn't expired).","commonSituations":"Re-draining the same worker shortly after a previous drain; a hung worker process that drained its functions but never shut down cleanly; stale membership entries in the metadata store.","solutions":["Wait for the worker to be removed from membership (check GET /admin/v2/worker/cluster) and retry later","Restart/stop the drained worker process so it deregisters cleanly","Investigate the worker's shutdown path if it stays registered indefinitely"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"boolean stillRegistered = admin.functions().getCluster().stream()\n    .anyMatch(w -> w.getWorkerId().equals(workerId)); // re-drain only after removal","typeGuard":null,"tryCatchPattern":"try {\n    admin.functions().drain(workerId);\n} catch (PulsarAdminException e) {\n    if (e.getStatusCode() == 412) {\n        // poll membership until the drained worker is removed, then retry\n    }\n}","preventionTips":["After a drain, confirm the worker left membership before re-draining","Ensure drained worker processes shut down cleanly and deregister","Watch for stale membership entries in the metadata store"],"tags":["rest-api","drain","membership"],"backgroundTag":"worker-still-registered","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}