weaviate/weaviate · warning

server is shutting down, recv stream closed after grace peri

Error message

server is shutting down, recv stream closed after grace period: %w

What it means

During server shutdown the batch-stream receiver checks, on every loop iteration before blocking on receive, whether the grace period timer (75 seconds, SHUTDOWN_GRACE_PERIOD) has expired while the client is still sending. If so, it cancels the stream context and returns this error, wrapping ctx.Err(), to force-close a stream whose client kept pushing messages after the shutdown signal. It protects the drain sequence from a misbehaving or slow client holding shutdown hostage.

Source

Thrown at adapters/handlers/grpc/v1/batch/stream.go:493

	var gracePeriod <-chan time.Time

	reqCh, errCh := h.recv(ctx, stream)
	for {
		// we must check for shutting down before we start blocking on h.recv in the event
		// that the client is misbehaving by sending more messages after the shutdown signal
		if h.shuttingDownCtx.Err() != nil {
			shuttingDownDone = nil // only do this once
			if gracePeriod == nil {
				// if we haven't already started the grace period timer then do so now
				gracePeriod = time.After(SHUTDOWN_GRACE_PERIOD)
				log.Info("server is shutting down, will force close recv stream after grace period")
			}
			select {
			case <-gracePeriod:
				// if we're still looping after the grace period has expired then force close
				log.Warn("grace period expired, closing recv stream")
				cancel()
				return fmt.Errorf("server is shutting down, recv stream closed after grace period: %w", ctx.Err())
			default:
				// otherwise continue as normal
			}
		}

		var request *pb.BatchStreamRequest
		var err error
		// non-blocking select to receive messages from the stream
		// this allows us to detect hanging clients during server shutdown
		// we either receive a request, an error, or a shutdown signal
		// if we receive a shutdown signal, we set up a grace period timer
		// after which we will force close the stream if it hasn't closed already
		// if we receive a request or an error, we process it as normal
		// if the context is cancelled, we exit the loop
		var ok bool
		select {
		case request, ok = <-reqCh:
			if !ok {

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Handle the shutting-down reply message in the client: stop sending, send the stop message, and close the stream gracefully.
2. Reconnect to another node and resume the import; unacked objects were rejected and must be resent.
3. Keep batch messages small enough that the server can drain them within the 75s grace period.
4. Upgrade the client to one that honors shutdown backoff signals during streaming batch imports.

Example fix

// client: react to shutting-down message
reply := <-replies
if reply.GetShutdownTriggered() != nil {
    sendStopMessage(stream)
    stream.CloseSend()
    return resumeOnOtherNode(pendingObjects)
}

// detect shutdown errors and resume elsewhere
if strings.Contains(err.Error(), "server is shutting down") || status.Code(err) == codes.Unavailable {
    // resend all unacked UUIDs to another node
}

When it happens

Trigger: A client continues sending BatchStreamRequest data messages after receiving the server's shutting-down signal, and the stream is still open when the 75s grace period expires; checked on the pre-block path before entering the non-blocking select.

Common situations: Client SDK versions that don't handle the `BatchStreamReply` shutting-down message and keep importing at full speed; very large in-flight batches that take longer than 75s to ack; long-lived import streams left open across a restart.

Related errors

• not accepting new streams: %w
• get classes for vectorisation check: %w
• invalid batch send request: data field is nil
• no results for generative search despite a search request. I
• select property is not an object

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/b67f4ca5f3f0e5dc. Report an issue: GitHub.