{"record":{"id":"daa752e9cea497b0","repo":"dapr/dapr","slug":"server-is-closed-daa752","errorCode":null,"errorMessage":"server is closed","messagePattern":"server is closed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/operator/api/httpendpoints.go","lineNumber":135,"sourceCode":"\t\t}\n\n\t\tb, err := json.Marshal(e)\n\t\tif err != nil {\n\t\t\tlog.Warnf(\"Error unmarshalling http endpoints: %s\", err)\n\t\t\tcontinue\n\t\t}\n\t\tresp.HttpEndpoints = append(resp.GetHttpEndpoints(), b)\n\t}\n\n\treturn resp, nil\n}\n\n// HTTPEndpointUpdate handles HTTP endpoint update streaming for a connected client.\n// Each client connection gets its own client loop that watches the informer\n// and sends updates over the gRPC stream.\nfunc (a *apiServer) HTTPEndpointUpdate(in *operatorv1pb.HTTPEndpointUpdateRequest, srv operatorv1pb.Operator_HTTPEndpointUpdateServer) error { //nolint:nosnakecase\n\tif a.closed.Load() {\n\t\treturn errors.New(\"server is closed\")\n\t}\n\n\tlog.Info(\"sidecar connected for http endpoint updates\")\n\n\tctx := srv.Context()\n\n\t// Verify authorization via informer's WatchUpdates, which checks SPIFFE ID\n\tch, cancel, err := a.endpointInformer.WatchUpdates(ctx, in.GetNamespace())\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tstream, err := sender.New(srv)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Create a client for this connection","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/dapr/dapr/blob/74ad41702745709bb15fe2114ff693b8c59bc3cc/pkg/operator/api/httpendpoints.go#L117-L153","documentation":"Returned by the operator's HTTPEndpointUpdate streaming handler when the apiServer has begun shutting down (the atomic 'closed' flag set at pkg/operator/api/api.go:166). It rejects new HTTP-endpoint update streams during the shutdown window before the gRPC listener stops accepting connections.","triggerScenarios":"A sidecar calls Operator_HTTPEndpointUpdate while the operator's run context is cancelled — e.g. during pod termination the gRPC server is still draining but new streams are already refused by the closed check at the top of the handler.","commonSituations":"Operator upgrades (rolling restart), operator crash/OOMKill with sidecars mid-reconnect, or Service endpoint propagation lag where a sidecar dials a dying replica.","solutions":["Rely on the sidecar's reconnect loop — this error is expected during shutdown and clears once a healthy replica answers.","Check operator pod health and restart count (kubectl get pods -n dapr-system) if it repeats continuously.","Stagger operator rolling updates so at least one replica stays ready.","Inspect the operator terminationGracePeriod if streams hang rather than fail fast."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := streamHttpEndpointUpdates(ctx)\nif err != nil && strings.Contains(err.Error(), \"server is closed\") {\n\t// operator drain window — reconnect to a healthy replica\n\treturn reconnectWithBackoff(ctx)\n}","preventionTips":["Rely on Dapr's sidecar watch loops which already reconnect; don't build one-shot clients for update streams.","Keep operator deployments healthy during workloads that consume HttpEndpoints (avoid all-replica restarts).","Alert on sustained reconnect storms, not on individual 'server is closed' errors."],"tags":["grpc","operator","shutdown","kubernetes","http-endpoints","streaming"],"backgroundTag":null,"analyzedSha":"74ad41702745709bb15fe2114ff693b8c59bc3cc","analyzedAt":"2026-08-16T04:22:26.543Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}