{"record":{"id":"d42dc763395dec8a","repo":"tailscale/tailscale","slug":"error-connecting-to-session-recorders-v-failure","errorCode":null,"errorMessage":"error connecting to session recorders: %v; failure mode is 'fail closed'; closing connection.","messagePattern":"error connecting to session recorders: (.+?); failure mode is 'fail closed'; closing connection\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"k8s-operator/sessionrecording/hijacker.go","lineNumber":178,"sourceCode":"\tqp := h.req.URL.Query()\n\tcontainer := strings.Join(qp[containerKey], \"\")\n\tvar recorderAddr net.Addr\n\ttrace := &httptrace.ClientTrace{\n\t\tGotConn: func(info httptrace.GotConnInfo) {\n\t\t\trecorderAddr = info.Conn.RemoteAddr()\n\t\t},\n\t}\n\twc, _, errChan, err = h.connectToRecorder(httptrace.WithClientTrace(ctx, trace), h.addrs, h.ts.Dial)\n\tif err != nil {\n\t\tmsg := fmt.Sprintf(\"error connecting to session recorders: %v\", err)\n\t\tif h.failOpen {\n\t\t\tmsg = msg + \"; failure mode is 'fail open'; continuing session without recording.\"\n\t\t\th.log.Warnf(msg)\n\t\t\treturn conn, nil\n\t\t}\n\t\tmsg = msg + \"; failure mode is 'fail closed'; closing connection.\"\n\t\tif err := closeConnWithWarning(conn, msg); err != nil {\n\t\t\treturn nil, errors.Join(errors.New(msg), err)\n\t\t}\n\t\treturn nil, errors.New(msg)\n\t} else {\n\t\th.log.Infof(\"%s session to container %q in Pod %q namespace %q will be recorded, the recording will be sent to a tsrecorder instance at %q\", h.sessionType, container, h.pod, h.ns, recorderAddr)\n\t}\n\n\tcl := tstime.DefaultClock{}\n\trec := tsrecorder.New(wc, cl, cl.Now(), h.failOpen, h.log)\n\ttty := strings.Join(qp[ttyKey], \"\")\n\thasTerm := (tty == \"true\") // session has terminal attached\n\tch := sessionrecording.CastHeader{\n\t\tVersion:   asciicastv2,\n\t\tTimestamp: cl.Now().Unix(),\n\t\tCommand:   strings.Join(qp[commandKey], \" \"),\n\t\tSrcNode:   strings.TrimSuffix(h.who.Node.Name, \".\"),\n\t\tSrcNodeID: h.who.Node.StableID,\n\t\tKubernetes: &sessionrecording.Kubernetes{\n\t\t\tPodName:     h.pod,","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/tailscale/tailscale/blob/cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042/k8s-operator/sessionrecording/hijacker.go#L160-L196","documentation":"Returned by the Kubernetes operator's session-recording proxy hijacker (k8s-operator/sessionrecording/hijacker.go:178) when connectToRecorder failed for all recorder addresses and the proxy is configured fail-closed (h.failOpen == false). Because recording is mandatory in this mode, the proxied exec/attach connection is torn down after attempting to send the warning message to the client. The returned error is the composed log-style message including the underlying dial error.","triggerScenarios":"A kubectl exec/cp/attach request on a ProxyGroup with session recording configured (recording: true) and failureMode fail-closed (the default) while no tsrecorder instance is reachable: wrong recorder addresses, tsrecorder pod down, NetworkPolicy/firewall blocking, TLS/SNI mismatch, or DNS for the recorder headless service failing.","commonSituations":"tsrecorder deployment scaled to zero or crashed; proxy class pointing at stale recorder FQDNs; cert rotation on tsrecorder making connections fail; cluster network policy added after initial setup; first-time install where recorder was never verified.","solutions":["Verify tsrecorder is running and the proxy can reach it: kubectl get pods for tsrecorder and test TLS to the address in the ProxyGroup recording config.","Fix the recorder addresses/TLS config in the ProxyGroup/proxy-class spec and ensure DNS resolves them from the operator/proxy pod.","Relax network policies/firewall rules so the operator can dial the recorder port.","If unrecorded sessions are acceptable in your threat model, set failureMode to fail-open (the source then logs and continues without recording) — an explicit risk decision."],"exampleFix":"# before: recorder down/unreachable, fail-closed kills every exec\nrecording: true # failureMode defaults to fail-closed\n# after: keep fail-closed but make the recorder reachable (preferred)\nkubectl -n tsrecorder get pods # must be Ready\nkubectl -n tailscale exec deploy/proxy -- wget -q -O- https://<recorder>:443/healthz\n# only if unrecorded sessions are acceptable:\n# set failureMode: fail-open in the ProxyGroup spec","handlingStrategy":"validation","validationCode":"// Health-check recorders before enabling mandatory recording (fail-closed).\nfor _, addr := range recorderAddrs {\n\tc := tls.Dialer{Timeout: 3 * time.Second}\n\tif conn, err := c.DialContext(ctx, \"tcp\", addr); err == nil {\n\t\tconn.Close()\n\t\tcontinue\n\t}\n\treturn fmt.Errorf(\"recorder %s unreachable; exec would be rejected in fail-closed mode\", addr)\n}","typeGuard":null,"tryCatchPattern":"if err := proxy.Hijack(...); err != nil {\n\tvar msg string\n\tif errors.As(err, &joined) { msg = joined.Error() } else { msg = err.Error() }\n\tif strings.Contains(msg, \"fail closed\") {\n\t\t// alert: recording is mandatory and recorders are down; fix recorder reachability\n\t}\n}","preventionTips":["Run tsrecorder with >=2 replicas and monitor its availability like a critical dependency.","Smoke-test recorder reachability (TLS + healthz) from the operator namespace on deploy.","Explicitly decide failureMode per environment; fail-closed means recorder outage blocks exec.","Alert on the 'error connecting to session recorders' log line."],"tags":["kubernetes","session-recording","tsrecorder","proxy","fail-closed"],"backgroundTag":null,"analyzedSha":"cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042","analyzedAt":"2026-08-15T19:58:31.583Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}