What causes "reader connection lost for container %s" in crowdsecurity/crowdsec?

Let it recover: TailContainer reconnects via backoff — check subsequent log lines for "connected to container logs".

reader connection lost for container %s

Error message

reader connection lost for container %s

What it means

Within tailContainerAttempt's read loop, when the readerTomb dies (the log stream reader terminated unexpectedly) the function kills the stream and returns "reader connection lost for container %s". Per the code comment, the known trigger is temporarily losing the connection to the docker socket — typically docker-socket-proxy dropping the connection or a docker daemon restart — not an application-level log error.

Source

Thrown at pkg/acquisition/modules/docker/run.go:683

			l.Process = true
			l.Module = d.GetName()
			evt := pipeline.MakeEvent(d.Config.UseTimeMachine, pipeline.LOG, true)
			evt.Line = l

			if d.metricsLevel != metrics.AcquisitionMetricsLevelNone {
				metrics.DockerDatasourceLinesRead.With(prometheus.Labels{"source": container.Name, "datasource_type": ModuleName, "acquis_type": evt.Line.Labels["type"]}).Inc()
			}

			outChan <- evt

			d.logger.Debugf("Sent line to parsing: %+v", evt.Line.Raw)
		case <-readerTomb.Dying():
			// This case is to handle temporarily losing the connection to the docker socket
			// The only known case currently is when using docker-socket-proxy (and maybe a docker daemon restart)
			container.logger.Debugf("readerTomb dying, connection lost")
			readerTomb.Kill(nil)

			return fmt.Errorf("reader connection lost for container %s", container.Name)
		}
	}
}

func (d *Source) TailService(ctx context.Context, service *ContainerConfig, outChan chan pipeline.Event, deleteChan chan *ContainerConfig) error {
	service.logger.Info("start monitoring")

	// we'll use just the interval generator, won't call backoff.Retry()
	bo := d.backoffFactory()
	firstRetry := true

	for {
		err := d.tailServiceAttempt(ctx, service, outChan, bo)
		if err == nil {
			// Successful completion - service was stopped gracefully
			return nil
		}

View on GitHub (pinned to 909b515798)

Solutions

Let it recover: TailContainer reconnects via backoff — check subsequent log lines for "connected to container logs".
Increase idle/keepalive timeouts on docker-socket-proxy or the TCP path to avoid recurring drops.
Pin docker_host to a local unix socket instead of an unreliable TCP/proxy hop.
If drops are constant, check `journalctl -u docker` for daemon restarts and fix the underlying instability.

Example fix

// before
docker_host: tcp://proxy:2375   // proxy drops idle streams
// after
docker_host: unix:///var/run/docker.sock

Defensive patterns

Strategy: retry

Try / catch

err := d.TailContainer(ctx, container, outChan, deleteChan)
if err != nil && strings.Contains(err.Error(), "reader connection lost") {
	// transient: rely on backoff reconnection, alert only if it repeats
	metrics.ReconnectCount.WithLabelValues(container.Name).Inc()
}

Prevention

Prefer a local unix socket over TCP/proxied docker hosts.
Tune docker-socket-proxy / LB idle timeouts above your log-stream inactivity.
Enable TCP keepalives on the docker host connection.
Alert on repeated reconnection loops rather than single drops.

When it happens

Trigger: The underlying HTTP stream to the docker daemon is cut mid-read: docker daemon restart, docker-socket-proxy connection reset/idle timeout, network interruption when using a TCP docker host. TailContainer then retries with backoff.

Common situations: Long-running crowdsec behind docker-socket-proxy with aggressive proxy timeouts; host docker upgrade/restart; system suspends; TCP docker_host behind a load balancer that kills idle streams.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/42e65e7e410ce6b1. Report an issue: GitHub.