{"record":{"id":"669846a431c020f7","repo":"weaviate/weaviate","slug":"worker-drain-timed-out-after-s","errorCode":null,"errorMessage":"worker drain timed out after %s","messagePattern":"worker drain timed out after (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/async_replication_scheduler.go","lineNumber":2066,"sourceCode":"\t\treturn yield()\n\t}\n\n\tif err := s.disableAsyncReplication(sched.ctx); err != nil {\n\t\treturn true, fail(\"stop\", err), false, false\n\t}\n\n\tif asyncRepRebuildAfterDisable != nil {\n\t\tasyncRepRebuildAfterDisable(s)\n\t}\n\n\t// Post-disable drain (normally instant — disable cancelled ctx + deregistered); a timeout\n\t// leaves the shard out of the mesh (enabling over a straggler risks a double-fold), so it\n\t// must be a loud failure with growing backoff, not a silent spin.\n\tselect {\n\tcase <-s.asyncRepDrained(sched.logger):\n\tcase <-time.After(time.Duration(asyncReplicationWorkerDrainTimeout.Load())):\n\t\tdrainTimeout := time.Duration(asyncReplicationWorkerDrainTimeout.Load())\n\t\treturn true, fail(\"drain\", fmt.Errorf(\"worker drain timed out after %s\", drainTimeout)), false, false\n\t}\n\n\t// Bail if Close() fired: enableAsyncReplication would otherwise spawn an\n\t// init-scan goroutine with no cancellable context.\n\tif sched.ctx.Err() != nil {\n\t\treturn false, 0, false, false\n\t}\n\n\tif err := s.enableAsyncReplication(sched.ctx, baseCfg); err != nil {\n\t\treturn true, fail(\"start\", err), false, false\n\t}\n\n\t// Reset backoff so subsequent height changes start clean.\n\ts.asyncRepRebuildFailures.Store(0)\n\ts.asyncRepRebuildBackoffUntil.Store(0)\n\n\t// Close() during enable registered against a cancelled scheduler, or a\n\t// teardown raced the enable. The authoritative cleanup is mayStopAsyncReplication","sourceCodeStart":2048,"sourceCodeEnd":2084,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/async_replication_scheduler.go#L2048-L2084","documentation":"Raised when waiting for async-replication workers to drain exceeds the configurable drain timeout (asyncReplicationWorkerDrainTimeout). The scheduler intentionally fails loudly (fail(\"drain\", ...)) with growing backoff instead of spinning silently, because a straggler shard must not be silently re-added to the replication mesh (risking a double-fold).","triggerScenarios":"Shard folding / async replication enablement waits on s.asyncRepDrained(); workers do not finish within the configured drain timeout, e.g. because a worker is stuck on a slow or unreachable remote replica.","commonSituations":"Slow network to a replica node, overloaded workers, very large shard backlogs during folding, or a drain timeout set too low for the workload.","solutions":["Increase asyncReplicationWorkerDrainTimeout to a value larger than your worst-case worker completion time","Check network connectivity and load on replica nodes holding up worker completion","Inspect replication worker logs/backoff to identify the straggler shard and repair or remove it","Retry the fold/enable operation after the straggler is healthy; the scheduler retries with growing backoff by design"],"exampleFix":"// before (timeout too small)\nasyncReplicationWorkerDrainTimeout.Store(int64(30 * time.Second))\n// after\nasyncReplicationWorkerDrainTimeout.Store(int64(5 * time.Minute))","handlingStrategy":"retry","validationCode":"// Before triggering fold/enable, check replica health\nfor _, n := range replicaNodes {\n    if !nodeHealthy(n) { return fmt.Errorf(\"replica %s unhealthy, postpone drain\", n) }\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"worker drain timed out\") {\n    // retry with backoff; the scheduler itself grows backoff\n    time.Sleep(backoff)\n    retry()\n}","preventionTips":["Size asyncReplicationWorkerDrainTimeout above worst-case replication latency","Monitor replica node load and network latency","Avoid folding shards while replicas are degraded"],"tags":["replication","timeout","shard-fold","drain"],"backgroundTag":"worker-drain-timeout","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}