What causes "writing to gzip writer: %w" in crowdsecurity/crowdsec?

Check host memory availability if this recurs — the in-memory gzip write failing usually means resource exhaustion

writing to gzip writer: %w

Error message

writing to gzip writer: %w

What it means

PrepareRequest gzips JSON bodies larger than 5KB and wraps any failure of the compressed write. gzip.Writer.Write only fails when the underlying bytes.Buffer write fails or the writer is in an error state — practically this is nearly impossible with an in-memory buffer, so this error indicates an unexpected internal failure during body compression.

Source

Thrown at pkg/apiclient/client_http.go:49

	var buf io.ReadWriter
	compressedBody := false

	if body != nil {
		jsonBuf := &bytes.Buffer{}
		enc := json.NewEncoder(jsonBuf)
		enc.SetEscapeHTML(false)

		if err = enc.Encode(body); err != nil {
			return nil, err
		}

		jsonBytes := jsonBuf.Bytes()
		if len(jsonBytes) > compressionMinSize {
			compressedBody = true
			buf = &bytes.Buffer{}
			gzipWriter := gzip.NewWriter(buf)
			if _, err = gzipWriter.Write(jsonBytes); err != nil {
				return nil, fmt.Errorf("writing to gzip writer: %w", err)
			}
			if err = gzipWriter.Close(); err != nil {
				return nil, fmt.Errorf("closing gzip writer: %w", err)
			}
		} else {
			buf = jsonBuf
		}
	}

	req, err := http.NewRequestWithContext(ctx, method, u.String(), buf)
	if err != nil {
		return nil, err
	}

	if body != nil {
		req.Header.Set("Content-Type", "application/json")
		if compressedBody {
			req.Header.Set("Content-Encoding", "gzip")

View on GitHub (pinned to 909b515798)

Solutions

Check host memory availability if this recurs — the in-memory gzip write failing usually means resource exhaustion
Retry the operation; if transient, a fresh call will succeed
Report to maintainers with the inner error if it reproduces reliably on a healthy host

Defensive patterns

Strategy: retry

Try / catch

for attempt := 0; attempt < 3; attempt++ {
    req, err := client.PrepareRequest(ctx, method, u, body)
    if err == nil {
        break
    }
    if strings.Contains(err.Error(), "gzip") {
        time.Sleep(time.Second) // transient compression/resource failure
        continue
    }
    return err
}

Prevention

Ensure hosts sending large signal batches have adequate memory
Keep individual request payloads reasonably sized
Treat gzip failures as infrastructural, not input problems

When it happens

Trigger: len(jsonBytes) > 5*1024 and gzipWriter.Write(jsonBytes) returns a non-nil error — e.g. an in-memory allocation failure or a writer corrupted by prior misuse (here a fresh writer, so essentially only OOM/resource-exhaustion conditions).

Common situations: Host under extreme memory pressure while serializing large alert/signal batches; custom instrumented builds that alter gzip behavior; otherwise effectively unreachable in normal operation.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/7f864c5ba80ad08f. Report an issue: GitHub.