{"record":{"id":"270605f1f1f232d6","repo":"crowdsecurity/crowdsec","slug":"reader-connection-lost-for-service-s","errorCode":null,"errorMessage":"reader connection lost for service %s","messagePattern":"reader connection lost for service (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/acquisition/modules/docker/run.go","lineNumber":801,"sourceCode":"\t\t\tl.Src = service.Name\n\t\t\tl.Process = true\n\t\t\tl.Module = d.GetName()\n\t\t\tevt := pipeline.MakeEvent(d.Config.UseTimeMachine, pipeline.LOG, true)\n\t\t\tevt.Line = l\n\n\t\t\tif d.metricsLevel != metrics.AcquisitionMetricsLevelNone {\n\t\t\t\tmetrics.DockerDatasourceLinesRead.With(prometheus.Labels{\"source\": service.Name, \"acquis_type\": l.Labels[\"type\"], \"datasource_type\": ModuleName}).Inc()\n\t\t\t}\n\n\t\t\toutChan <- evt\n\n\t\t\td.logger.Debugf(\"Sent line to parsing: %+v\", evt.Line.Raw)\n\t\tcase <-readerTomb.Dying():\n\t\t\t// Handle connection loss similar to containers\n\t\t\tservice.logger.Debugf(\"readerTomb dying, connection lost\")\n\t\t\treaderTomb.Kill(nil)\n\n\t\t\treturn fmt.Errorf(\"reader connection lost for service %s\", service.Name)\n\t\t}\n\t}\n}\n\nfunc (d *Source) ContainerManager(ctx context.Context, in chan *ContainerConfig, deleteChan chan *ContainerConfig, outChan chan pipeline.Event) error {\n\td.logger.Info(\"Container Manager started\")\n\n\tfor {\n\t\tselect {\n\t\tcase newContainer := <-in:\n\t\t\tif _, ok := d.runningContainerState.Get(newContainer.ID); !ok {\n\t\t\t\tnewContainer.logger = d.logger.WithField(\"container_name\", newContainer.Name)\n\t\t\t\tnewContainer.t.Go(func() error {\n\t\t\t\t\treturn d.TailContainer(ctx, newContainer, outChan, deleteChan)\n\t\t\t\t})\n\n\t\t\t\td.runningContainerState.Set(newContainer.ID, newContainer)\n\t\t\t}","sourceCodeStart":783,"sourceCodeEnd":819,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/docker/run.go#L783-L819","documentation":"Mirroring the container path, tailServiceAttempt returns \"reader connection lost for service %s\" when the readerTomb dies while streaming swarm service logs — the connection to the docker daemon dropped mid-stream. TailService then retries the attempt (with backoff reset on successful reconnection).","triggerScenarios":"Mid-read disconnection of the ServiceLogs stream: daemon restart, proxy timeout on the long-lived log stream, network interruption to the manager node, or service logs stream closed by the daemon.","commonSituations":"Swarm manager restarted/upgraded; LB between crowdsec and manager killing idle streams; proxy idle timeouts on streaming responses; large gaps between log lines causing idle-stream reaps.","solutions":["Check for automatic recovery — TailService reconnects; look for \"connected to service logs\" afterwards.","Raise proxy/LB idle timeouts for streaming endpoints (/services/{id}/logs).","Run crowdsec closer to the manager (unix socket) to remove the unreliable network hop.","Inspect manager stability (`docker service logs`, journalctl for dockerd restarts) if drops repeat."],"exampleFix":"// before\nstream_idle_timeout: 60s  # proxy kills idle log streams\n// after\nstream_idle_timeout: 1h","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := d.TailService(ctx, service, outChan, deleteChan)\nif err != nil && strings.Contains(err.Error(), \"reader connection lost\") {\n\t// transient stream drop: backoff reconnect, count and alert on recurrence\n\tmetrics.ReconnectCount.WithLabelValues(service.Name).Inc()\n}","preventionTips":["Keep crowdsec network-adjacent to the swarm manager (unix socket when co-located).","Raise streaming idle timeouts on proxies/LBs between crowdsec and the manager.","Track manager node restarts and correlate with source reconnects.","Ensure services emit logs regularly or expect idle-stream reaps; rely on backoff."],"tags":["docker","swarm","connection-lost","reconnect"],"backgroundTag":"broken-pipe","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}