dapr/dapr · warning
server is closed
Error message
server is closed
What it means
Returned by the operator's HTTPEndpointUpdate streaming handler when the apiServer has begun shutting down (the atomic 'closed' flag set at pkg/operator/api/api.go:166). It rejects new HTTP-endpoint update streams during the shutdown window before the gRPC listener stops accepting connections.
Source
Thrown at pkg/operator/api/httpendpoints.go:135
}
b, err := json.Marshal(e)
if err != nil {
log.Warnf("Error unmarshalling http endpoints: %s", err)
continue
}
resp.HttpEndpoints = append(resp.GetHttpEndpoints(), b)
}
return resp, nil
}
// HTTPEndpointUpdate handles HTTP endpoint update streaming for a connected client.
// Each client connection gets its own client loop that watches the informer
// and sends updates over the gRPC stream.
func (a *apiServer) HTTPEndpointUpdate(in *operatorv1pb.HTTPEndpointUpdateRequest, srv operatorv1pb.Operator_HTTPEndpointUpdateServer) error { //nolint:nosnakecase
if a.closed.Load() {
return errors.New("server is closed")
}
log.Info("sidecar connected for http endpoint updates")
ctx := srv.Context()
// Verify authorization via informer's WatchUpdates, which checks SPIFFE ID
ch, cancel, err := a.endpointInformer.WatchUpdates(ctx, in.GetNamespace())
if err != nil {
return err
}
stream, err := sender.New(srv)
if err != nil {
return err
}
// Create a client for this connectionView on GitHub (pinned to 74ad417027)
Solutions
- Rely on the sidecar's reconnect loop — this error is expected during shutdown and clears once a healthy replica answers.
- Check operator pod health and restart count (kubectl get pods -n dapr-system) if it repeats continuously.
- Stagger operator rolling updates so at least one replica stays ready.
- Inspect the operator terminationGracePeriod if streams hang rather than fail fast.
Defensive patterns
Strategy: retry
Try / catch
err := streamHttpEndpointUpdates(ctx)
if err != nil && strings.Contains(err.Error(), "server is closed") {
// operator drain window — reconnect to a healthy replica
return reconnectWithBackoff(ctx)
} Prevention
- Rely on Dapr's sidecar watch loops which already reconnect; don't build one-shot clients for update streams.
- Keep operator deployments healthy during workloads that consume HttpEndpoints (avoid all-replica restarts).
- Alert on sustained reconnect storms, not on individual 'server is closed' errors.
When it happens
Trigger: A sidecar calls Operator_HTTPEndpointUpdate while the operator's run context is cancelled — e.g. during pod termination the gRPC server is still draining but new streams are already refused by the closed check at the top of the handler.
Common situations: Operator upgrades (rolling restart), operator crash/OOMKill with sidecars mid-reconnect, or Service endpoint propagation lag where a sidecar dials a dying replica.
Related errors
AI-assisted analysis of dapr/dapr@74ad417027 (2026-08-16).
Data as JSON: /api/errors/daa752e9cea497b0.
Report an issue: GitHub.