{"record":{"id":"fe86afb67e536086","repo":"vitessio/vitess","slug":"grpc-server-stopped-w","errorCode":null,"errorMessage":"grpc server stopped: %w","messagePattern":"grpc server stopped: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtadmin/grpcserver/server.go","lineNumber":238,"sourceCode":"\t\tsig := <-signals\n\t\terr := fmt.Errorf(\"received signal: %v\", sig)\n\t\tlog.Warn(fmt.Sprint(err))\n\t\tshutdown <- err\n\t}()\n\n\tif s.opts.MetricsEndpoint != \"\" {\n\t\tif !strings.HasPrefix(s.opts.MetricsEndpoint, \"/\") {\n\t\t\ts.opts.MetricsEndpoint = \"/\" + s.opts.MetricsEndpoint\n\t\t}\n\n\t\tgrpc_prometheus.Register(s.gRPCServer)\n\t\ts.router.Handle(s.opts.MetricsEndpoint, promhttp.Handler())\n\t}\n\n\t// Start the servers\n\tgo func() {\n\t\terr := s.gRPCServer.Serve(grpcLis)\n\t\terr = fmt.Errorf(\"grpc server stopped: %w\", err)\n\t\tlog.Warn(fmt.Sprint(err))\n\t\tshutdown <- err\n\t}()\n\n\tgo func() {\n\t\terr := http.Serve(anyLis, s.router)\n\t\terr = fmt.Errorf(\"http server stopped: %w\", err)\n\t\tlog.Warn(fmt.Sprint(err))\n\t\tshutdown <- err\n\t}()\n\n\t// Start muxing connections\n\tgo func() {\n\t\terr := lmux.Serve()\n\t\terr = fmt.Errorf(\"listener closed: %w\", err)\n\t\tlog.Warn(fmt.Sprint(err))\n\t\tshutdown <- err\n\t}()","sourceCodeStart":220,"sourceCodeEnd":256,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/grpcserver/server.go#L220-L256","documentation":"This error is produced by vtadmin's gRPC server goroutine when `grpcServer.Serve()` returns, which only happens if the server stops unexpectedly (it blocks until failure or explicit Stop). The returned error is wrapped with 'grpc server stopped' and sent to the shutdown channel, which triggers the whole vtadmin server shutdown. It is a signal that the gRPC listener died, not necessarily a fatal bug — during graceful shutdown Serve returns a normal error.","triggerScenarios":"Calling grpcserver.Server.Serve() and the underlying gRPC server stops: the listener is closed by another component, Serve is called twice, the listener fails to accept connections, or the server is being shut down via Close/GracefulStop.","commonSituations":"Another goroutine closes the shared listener while vtadmin is running; port hijacked or listener error at OS level; vtadmin shutting down normally and this warning appearing in logs as part of the shutdown sequence.","solutions":["Check the wrapped inner error in the log to find the root cause (listener closed vs. Serve called twice).","If seen at shutdown, ignore it — it is the expected path when the process is terminating.","Ensure nothing in your setup closes the gRPC listener independently of grpcserver.Close().","Verify Serve() is only invoked once per grpcserver.Server instance."],"exampleFix":"// before\ngo func() {\n\tif err := srv.Serve(lis); err != nil {\n\t\tlog.Fatal(err)\n\t}\n}()\n// after\ngo func() {\n\terr := srv.Serve(lis)\n\tif err != nil && !errors.Is(err, grpc.ErrServerStopped) {\n\t\tlog.Warn(fmt.Sprintf(\"grpc server stopped: %v\", err))\n\t}\n\tshutdown <- err\n}()","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func isExpectedStop(err error) bool {\n\treturn err == nil || errors.Is(err, grpc.ErrServerStopped) || errors.Is(err, net.ErrClosed)\n}","tryCatchPattern":"err := <-shutdownCh\nif isExpectedStop(err) {\n\tlog.Info(\"vtadmin shutting down\")\n} else {\n\tlog.Error(fmt.Sprintf(\"unexpected server stop: %v\", err))\n}","preventionTips":["Do not close the gRPC listener outside the grpcserver Close path.","Call Serve exactly once per server instance.","Treat 'server stopped' warnings during shutdown as expected; alert only on unexpected stops.","Monitor the wrapped root cause, not the wrapper message."],"tags":["grpc","server-lifecycle","vtadmin"],"backgroundTag":"grpc-server-stopped","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}