crowdsecurity/crowdsec · error

error while reading tail response: %w

Error message

error while reading tail response: %w

What it means

The background goroutine that streams the tail HTTP response body into the channel hit an error while decoding server-sent-events/NDJSON from VictoriaLogs. This runs inside the tomb, so the failure kills the acquisition goroutine group rather than returning to the caller.

Source

Thrown at pkg/acquisition/modules/victorialogs/internal/vlclient/vl_client.go:349

		break
	}

	if resp.StatusCode != http.StatusOK {
		lc.Logger.Warnf("bad HTTP response code for tail request: %d", resp.StatusCode)
		body, _ := io.ReadAll(resp.Body)
		resp.Body.Close()

		if ok := lc.shouldRetry(); !ok {
			return nil, fmt.Errorf("bad HTTP response code: %d: %s: %w", resp.StatusCode, string(body), err)
		}
	}

	responseChan := make(chan *Log)

	lc.t.Go(func() error {
		_, _, err = lc.readResponse(ctx, resp, responseChan)
		if err != nil {
			return fmt.Errorf("error while reading tail response: %w", err)
		}

		return nil
	})

	return responseChan, nil
}

// QueryRange queries the logs
// See: https://docs.victoriametrics.com/victorialogs/querying/#querying-logs
func (lc *VLClient) QueryRange(ctx context.Context, infinite bool) chan *Log {
	t := time.Now().Add(-1 * lc.config.Since)
	u := lc.getURLFor("select/logsql/query", map[string]string{
		"query": lc.config.Query,
		"start": t.Format(time.RFC3339Nano),
		"limit": strconv.Itoa(lc.config.Limit),
	})

View on GitHub (pinned to 909b515798)

Solutions

1. Check the wrapped inner error: if it is `unexpected EOF`/`connection reset`, suspect intermediaries and raise their idle/proxy timeouts.
2. Verify the VictoriaLogs version is compatible; upgrade the datasource or crowdsec if the payload format changed.
3. Ensure stable network between crowdsec and VictoriaLogs for long-lived streaming connections.
4. If timeouts are the cause, adjust client timeout settings for the streaming request.
5. Restart crowdsec; in TAIL_MODE the stream must be re-established.

Example fix

// nginx-style intermediary
// before
proxy_read_timeout 30s;
// after
proxy_read_timeout 3600s;  # keep long-lived tail streams alive

// in the tomb goroutine
if err := readResponse(ctx, resp, ch); err != nil {
    if errors.Is(err, io.EOF) || errors.Is(err, context.Canceled) {
        return nil // clean end / shutdown
    }
    return fmt.Errorf("error while reading tail response: %w", err)
}

When it happens

Trigger: `readResponse` fails mid-stream: the tail connection is dropped by the server or a proxy, a read timeout fires, the body contains malformed/undecodable log entries, or ctx is canceled causing the response read to abort with an unexpected error.

Common situations: A load balancer with a short idle timeout kills long-lived tail connections; network interruption during live tailing; VictoriaLogs crashes mid-response; incompatible VictoriaLogs version emitting a payload shape this client cannot decode.

Related errors

• cannot read line in response: %w
• while reading %s/%s: %w
• cannot get records: %w
• querying range: %w
• error tailing logs: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/816e00c7d17187ca. Report an issue: GitHub.