{"record":{"id":"42e65e7e410ce6b1","repo":"crowdsecurity/crowdsec","slug":"reader-connection-lost-for-container-s","errorCode":null,"errorMessage":"reader connection lost for container %s","messagePattern":"reader connection lost for container (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/acquisition/modules/docker/run.go","lineNumber":683,"sourceCode":"\t\t\tl.Process = true\n\t\t\tl.Module = d.GetName()\n\t\t\tevt := pipeline.MakeEvent(d.Config.UseTimeMachine, pipeline.LOG, true)\n\t\t\tevt.Line = l\n\n\t\t\tif d.metricsLevel != metrics.AcquisitionMetricsLevelNone {\n\t\t\t\tmetrics.DockerDatasourceLinesRead.With(prometheus.Labels{\"source\": container.Name, \"datasource_type\": ModuleName, \"acquis_type\": evt.Line.Labels[\"type\"]}).Inc()\n\t\t\t}\n\n\t\t\toutChan <- evt\n\n\t\t\td.logger.Debugf(\"Sent line to parsing: %+v\", evt.Line.Raw)\n\t\tcase <-readerTomb.Dying():\n\t\t\t// This case is to handle temporarily losing the connection to the docker socket\n\t\t\t// The only known case currently is when using docker-socket-proxy (and maybe a docker daemon restart)\n\t\t\tcontainer.logger.Debugf(\"readerTomb dying, connection lost\")\n\t\t\treaderTomb.Kill(nil)\n\n\t\t\treturn fmt.Errorf(\"reader connection lost for container %s\", container.Name)\n\t\t}\n\t}\n}\n\nfunc (d *Source) TailService(ctx context.Context, service *ContainerConfig, outChan chan pipeline.Event, deleteChan chan *ContainerConfig) error {\n\tservice.logger.Info(\"start monitoring\")\n\n\t// we'll use just the interval generator, won't call backoff.Retry()\n\tbo := d.backoffFactory()\n\tfirstRetry := true\n\n\tfor {\n\t\terr := d.tailServiceAttempt(ctx, service, outChan, bo)\n\t\tif err == nil {\n\t\t\t// Successful completion - service was stopped gracefully\n\t\t\treturn nil\n\t\t}\n","sourceCodeStart":665,"sourceCodeEnd":701,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/docker/run.go#L665-L701","documentation":"Within tailContainerAttempt's read loop, when the readerTomb dies (the log stream reader terminated unexpectedly) the function kills the stream and returns \"reader connection lost for container %s\". Per the code comment, the known trigger is temporarily losing the connection to the docker socket — typically docker-socket-proxy dropping the connection or a docker daemon restart — not an application-level log error.","triggerScenarios":"The underlying HTTP stream to the docker daemon is cut mid-read: docker daemon restart, docker-socket-proxy connection reset/idle timeout, network interruption when using a TCP docker host. TailContainer then retries with backoff.","commonSituations":"Long-running crowdsec behind docker-socket-proxy with aggressive proxy timeouts; host docker upgrade/restart; system suspends; TCP docker_host behind a load balancer that kills idle streams.","solutions":["Let it recover: TailContainer reconnects via backoff — check subsequent log lines for \"connected to container logs\".","Increase idle/keepalive timeouts on docker-socket-proxy or the TCP path to avoid recurring drops.","Pin docker_host to a local unix socket instead of an unreliable TCP/proxy hop.","If drops are constant, check `journalctl -u docker` for daemon restarts and fix the underlying instability."],"exampleFix":"// before\ndocker_host: tcp://proxy:2375   // proxy drops idle streams\n// after\ndocker_host: unix:///var/run/docker.sock","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := d.TailContainer(ctx, container, outChan, deleteChan)\nif err != nil && strings.Contains(err.Error(), \"reader connection lost\") {\n\t// transient: rely on backoff reconnection, alert only if it repeats\n\tmetrics.ReconnectCount.WithLabelValues(container.Name).Inc()\n}","preventionTips":["Prefer a local unix socket over TCP/proxied docker hosts.","Tune docker-socket-proxy / LB idle timeouts above your log-stream inactivity.","Enable TCP keepalives on the docker host connection.","Alert on repeated reconnection loops rather than single drops."],"tags":["docker","logs","connection-lost","reconnect"],"backgroundTag":"broken-pipe","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}