{"record":{"id":"4fc3256624df97e2","repo":"apache/pulsar","slug":"another-drain-is-in-progress-4fc325","errorCode":null,"errorMessage":"Another drain is in progress","messagePattern":"Another drain is in progress","errorType":"http","errorClass":"RestException","httpStatus":409,"severity":"warning","filePath":"pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java","lineNumber":274,"sourceCode":"                .attr(\"workerId\", workerId)\n                .attr(\"clientRole\", authParams.getClientRole())\n                .attr(\"originalPrincipal\", authParams.getOriginalPrincipal())\n                .attr(\"calledOnLeaderUri\", calledOnLeaderUri)\n                .attr(\"actualWorkerId\", actualWorkerId)\n                .log(\"drain called\");\n\n        throwIfNotSuperUser(authParams, \"drain worker\");\n\n        // Depending on which operations we decide to allow, we may add checks here to error/exception if\n        //      calledOnLeaderUri is true on a non-leader\n        //      calledOnLeaderUri is false on a leader\n        // For now, deal with everything.\n\n        if (worker().getLeaderService().isLeader()) {\n            try {\n                worker().getSchedulerManager().drainIfNotInProgress(workerId);\n            } catch (SchedulerManager.DrainInProgressException e) {\n                throw new RestException(Status.CONFLICT, \"Another drain is in progress\");\n            } catch (SchedulerManager.TooFewWorkersException e) {\n                throw new RestException(Status.BAD_REQUEST, \"Too few workers (need at least 2)\");\n            } catch (SchedulerManager.WorkerNotRemovedAfterPriorDrainException e) {\n                String errString = \"Worker \" + workerId + \" was not yet removed after a prior drain op; try later\";\n                throw new RestException(Status.PRECONDITION_FAILED, errString);\n            } catch (SchedulerManager.UnknownWorkerException e) {\n                String errString = \"Worker \" + workerId + \" is not among the current workers in the system\";\n                throw new RestException(Status.BAD_REQUEST, errString);\n            }\n        } else {\n            URI redirect = buildRedirectUriForDrainRelatedOp(uri, workerId);\n            log.info().attr(\"redirect\", redirect).log(\"Not leader; redirect URI=\");\n            throw new WebApplicationException(Response.temporaryRedirect(redirect).build());\n        }\n    }\n\n    @Override\n    public LongRunningProcessStatus getDrainStatus(final URI uri, final String inWorkerId,","sourceCodeStart":256,"sourceCodeEnd":292,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java#L256-L292","documentation":"This HTTP 409 error is returned by the drain endpoint when a drain operation is already running. SchedulerManager.drainIfNotInProgress() enforces a single concurrent drain cluster-wide to avoid conflicting function-instance migrations while a worker is being decommissioned.","triggerScenarios":"Calling PUT /admin/v2/worker/drain/{workerId} (or pulsar-admin functions worker drain) while a prior drain on the leader is still in progress.","commonSituations":"Draining multiple workers back-to-back without waiting for each drain to finish; automation that retries drain on timeout; large function counts making drains slow.","solutions":["Wait for the current drain to complete before starting another","Poll the drain/status endpoint or watch worker logs for drain completion","Serialize drain requests in automation (no concurrent drain calls)"],"exampleFix":"// before: fire-and-forget loop\nfor (String w : workers) { admin.functions().drain(w); }\n// after: sequential with conflict handling\nfor (String w : workers) {\n    try {\n        admin.functions().drain(w);\n    } catch (PulsarAdminException e) {\n        if (e.getStatusCode() == 409) { /* wait for in-progress drain, retry */ }\n    }\n}","handlingStrategy":"try-catch","validationCode":"// no pre-check API for in-progress drain; serialize drains in your tooling","typeGuard":null,"tryCatchPattern":"try {\n    admin.functions().drain(workerId);\n} catch (PulsarAdminException e) {\n    if (e.getStatusCode() == 409 && e.getMessage().contains(\"Another drain is in progress\")) {\n        // wait for current drain to finish, then retry\n    }\n}","preventionTips":["Never issue concurrent drain requests","Wait for each drain to complete before draining the next worker","Add retry-with-backoff handling for 409 responses in automation"],"tags":["rest-api","concurrency","drain"],"backgroundTag":"drain-in-progress","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}