weaviate/weaviate · error

worker drain timed out after %s

Error message

worker drain timed out after %s

What it means

Raised when waiting for async-replication workers to drain exceeds the configurable drain timeout (asyncReplicationWorkerDrainTimeout). The scheduler intentionally fails loudly (fail("drain", ...)) with growing backoff instead of spinning silently, because a straggler shard must not be silently re-added to the replication mesh (risking a double-fold).

Source

Thrown at adapters/repos/db/async_replication_scheduler.go:2066

		return yield()
	}

	if err := s.disableAsyncReplication(sched.ctx); err != nil {
		return true, fail("stop", err), false, false
	}

	if asyncRepRebuildAfterDisable != nil {
		asyncRepRebuildAfterDisable(s)
	}

	// Post-disable drain (normally instant — disable cancelled ctx + deregistered); a timeout
	// leaves the shard out of the mesh (enabling over a straggler risks a double-fold), so it
	// must be a loud failure with growing backoff, not a silent spin.
	select {
	case <-s.asyncRepDrained(sched.logger):
	case <-time.After(time.Duration(asyncReplicationWorkerDrainTimeout.Load())):
		drainTimeout := time.Duration(asyncReplicationWorkerDrainTimeout.Load())
		return true, fail("drain", fmt.Errorf("worker drain timed out after %s", drainTimeout)), false, false
	}

	// Bail if Close() fired: enableAsyncReplication would otherwise spawn an
	// init-scan goroutine with no cancellable context.
	if sched.ctx.Err() != nil {
		return false, 0, false, false
	}

	if err := s.enableAsyncReplication(sched.ctx, baseCfg); err != nil {
		return true, fail("start", err), false, false
	}

	// Reset backoff so subsequent height changes start clean.
	s.asyncRepRebuildFailures.Store(0)
	s.asyncRepRebuildBackoffUntil.Store(0)

	// Close() during enable registered against a cancelled scheduler, or a
	// teardown raced the enable. The authoritative cleanup is mayStopAsyncReplication

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Increase asyncReplicationWorkerDrainTimeout to a value larger than your worst-case worker completion time
2. Check network connectivity and load on replica nodes holding up worker completion
3. Inspect replication worker logs/backoff to identify the straggler shard and repair or remove it
4. Retry the fold/enable operation after the straggler is healthy; the scheduler retries with growing backoff by design

Example fix

// before (timeout too small)
asyncReplicationWorkerDrainTimeout.Store(int64(30 * time.Second))
// after
asyncReplicationWorkerDrainTimeout.Store(int64(5 * time.Minute))

// Before triggering fold/enable, check replica health
for _, n := range replicaNodes {
    if !nodeHealthy(n) { return fmt.Errorf("replica %s unhealthy, postpone drain", n) }
}

if err != nil && strings.Contains(err.Error(), "worker drain timed out") {
    // retry with backoff; the scheduler itself grows backoff
    time.Sleep(backoff)
    retry()
}

When it happens

Trigger: Shard folding / async replication enablement waits on s.asyncRepDrained(); workers do not finish within the configured drain timeout, e.g. because a worker is stuck on a slow or unreachable remote replica.

Common situations: Slow network to a replica node, overloaded workers, very large shard backlogs during folding, or a drain timeout set too low for the workload.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• gRPC PutObjects: %w
• gRPC MergeObject: %w
• gRPC DeleteObject: %w
• gRPC DeleteObjects: %w
• gRPC AddReferences: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/669846a431c020f7. Report an issue: GitHub.