{"record":{"id":"b67f4ca5f3f0e5dc","repo":"weaviate/weaviate","slug":"server-is-shutting-down-recv-stream-closed-after","errorCode":null,"errorMessage":"server is shutting down, recv stream closed after grace period: %w","messagePattern":"server is shutting down, recv stream closed after grace period: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"adapters/handlers/grpc/v1/batch/stream.go","lineNumber":493,"sourceCode":"\tvar gracePeriod <-chan time.Time\n\n\treqCh, errCh := h.recv(ctx, stream)\n\tfor {\n\t\t// we must check for shutting down before we start blocking on h.recv in the event\n\t\t// that the client is misbehaving by sending more messages after the shutdown signal\n\t\tif h.shuttingDownCtx.Err() != nil {\n\t\t\tshuttingDownDone = nil // only do this once\n\t\t\tif gracePeriod == nil {\n\t\t\t\t// if we haven't already started the grace period timer then do so now\n\t\t\t\tgracePeriod = time.After(SHUTDOWN_GRACE_PERIOD)\n\t\t\t\tlog.Info(\"server is shutting down, will force close recv stream after grace period\")\n\t\t\t}\n\t\t\tselect {\n\t\t\tcase <-gracePeriod:\n\t\t\t\t// if we're still looping after the grace period has expired then force close\n\t\t\t\tlog.Warn(\"grace period expired, closing recv stream\")\n\t\t\t\tcancel()\n\t\t\t\treturn fmt.Errorf(\"server is shutting down, recv stream closed after grace period: %w\", ctx.Err())\n\t\t\tdefault:\n\t\t\t\t// otherwise continue as normal\n\t\t\t}\n\t\t}\n\n\t\tvar request *pb.BatchStreamRequest\n\t\tvar err error\n\t\t// non-blocking select to receive messages from the stream\n\t\t// this allows us to detect hanging clients during server shutdown\n\t\t// we either receive a request, an error, or a shutdown signal\n\t\t// if we receive a shutdown signal, we set up a grace period timer\n\t\t// after which we will force close the stream if it hasn't closed already\n\t\t// if we receive a request or an error, we process it as normal\n\t\t// if the context is cancelled, we exit the loop\n\t\tvar ok bool\n\t\tselect {\n\t\tcase request, ok = <-reqCh:\n\t\t\tif !ok {","sourceCodeStart":475,"sourceCodeEnd":511,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/handlers/grpc/v1/batch/stream.go#L475-L511","documentation":"During server shutdown the batch-stream receiver checks, on every loop iteration before blocking on receive, whether the grace period timer (75 seconds, SHUTDOWN_GRACE_PERIOD) has expired while the client is still sending. If so, it cancels the stream context and returns this error, wrapping ctx.Err(), to force-close a stream whose client kept pushing messages after the shutdown signal. It protects the drain sequence from a misbehaving or slow client holding shutdown hostage.","triggerScenarios":"A client continues sending BatchStreamRequest data messages after receiving the server's shutting-down signal, and the stream is still open when the 75s grace period expires; checked on the pre-block path before entering the non-blocking select.","commonSituations":"Client SDK versions that don't handle the `BatchStreamReply` shutting-down message and keep importing at full speed; very large in-flight batches that take longer than 75s to ack; long-lived import streams left open across a restart.","solutions":["Handle the shutting-down reply message in the client: stop sending, send the stop message, and close the stream gracefully.","Reconnect to another node and resume the import; unacked objects were rejected and must be resent.","Keep batch messages small enough that the server can drain them within the 75s grace period.","Upgrade the client to one that honors shutdown backoff signals during streaming batch imports."],"exampleFix":"// client: react to shutting-down message\nreply := <-replies\nif reply.GetShutdownTriggered() != nil {\n    sendStopMessage(stream)\n    stream.CloseSend()\n    return resumeOnOtherNode(pendingObjects)\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// detect shutdown errors and resume elsewhere\nif strings.Contains(err.Error(), \"server is shutting down\") || status.Code(err) == codes.Unavailable {\n    // resend all unacked UUIDs to another node\n}","preventionTips":["Handle the BatchStreamReply shutting-down message: stop sending and half-close promptly.","Keep batch messages small so the server can drain within the 75s grace period.","Track acked UUIDs per message so an interrupted stream can resume without duplicates."],"tags":["grpc","shutdown","grace-period","batch-streaming"],"backgroundTag":"server-shutting-down","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}