What causes "reader connection lost for service %s" in crowdsecurity/crowdsec?

Check for automatic recovery — TailService reconnects; look for "connected to service logs" afterwards.

reader connection lost for service %s

Error message

reader connection lost for service %s

What it means

Mirroring the container path, tailServiceAttempt returns "reader connection lost for service %s" when the readerTomb dies while streaming swarm service logs — the connection to the docker daemon dropped mid-stream. TailService then retries the attempt (with backoff reset on successful reconnection).

Source

Thrown at pkg/acquisition/modules/docker/run.go:801

			l.Src = service.Name
			l.Process = true
			l.Module = d.GetName()
			evt := pipeline.MakeEvent(d.Config.UseTimeMachine, pipeline.LOG, true)
			evt.Line = l

			if d.metricsLevel != metrics.AcquisitionMetricsLevelNone {
				metrics.DockerDatasourceLinesRead.With(prometheus.Labels{"source": service.Name, "acquis_type": l.Labels["type"], "datasource_type": ModuleName}).Inc()
			}

			outChan <- evt

			d.logger.Debugf("Sent line to parsing: %+v", evt.Line.Raw)
		case <-readerTomb.Dying():
			// Handle connection loss similar to containers
			service.logger.Debugf("readerTomb dying, connection lost")
			readerTomb.Kill(nil)

			return fmt.Errorf("reader connection lost for service %s", service.Name)
		}
	}
}

func (d *Source) ContainerManager(ctx context.Context, in chan *ContainerConfig, deleteChan chan *ContainerConfig, outChan chan pipeline.Event) error {
	d.logger.Info("Container Manager started")

	for {
		select {
		case newContainer := <-in:
			if _, ok := d.runningContainerState.Get(newContainer.ID); !ok {
				newContainer.logger = d.logger.WithField("container_name", newContainer.Name)
				newContainer.t.Go(func() error {
					return d.TailContainer(ctx, newContainer, outChan, deleteChan)
				})

				d.runningContainerState.Set(newContainer.ID, newContainer)
			}

View on GitHub (pinned to 909b515798)

Solutions

Check for automatic recovery — TailService reconnects; look for "connected to service logs" afterwards.
Raise proxy/LB idle timeouts for streaming endpoints (/services/{id}/logs).
Run crowdsec closer to the manager (unix socket) to remove the unreliable network hop.
Inspect manager stability (`docker service logs`, journalctl for dockerd restarts) if drops repeat.

Example fix

// before
stream_idle_timeout: 60s  # proxy kills idle log streams
// after
stream_idle_timeout: 1h

Defensive patterns

Strategy: retry

Try / catch

err := d.TailService(ctx, service, outChan, deleteChan)
if err != nil && strings.Contains(err.Error(), "reader connection lost") {
	// transient stream drop: backoff reconnect, count and alert on recurrence
	metrics.ReconnectCount.WithLabelValues(service.Name).Inc()
}

Prevention

Keep crowdsec network-adjacent to the swarm manager (unix socket when co-located).
Raise streaming idle timeouts on proxies/LBs between crowdsec and the manager.
Track manager node restarts and correlate with source reconnects.
Ensure services emit logs regularly or expect idle-stream reaps; rely on backoff.

When it happens

Trigger: Mid-read disconnection of the ServiceLogs stream: daemon restart, proxy timeout on the long-lived log stream, network interruption to the manager node, or service logs stream closed by the daemon.

Common situations: Swarm manager restarted/upgraded; LB between crowdsec and manager killing idle streams; proxy idle timeouts on streaming responses; large gaps between log lines causing idle-stream reaps.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/270605f1f1f232d6. Report an issue: GitHub.