crowdsecurity/crowdsec · error

while validating reader configuration: %w

Error message

while validating reader configuration: %w

What it means

Wraps the error from kafka-go's ReaderConfig.Validate() when building the kafka reader for the datasource. Validate enforces kafka-go's structural rules on the config (e.g. Brokers must not be empty, MinBytes/MaxBytes ordering, GroupID constraints). This surfaces those library-level constraints as part of CrowdSec's kafka source configuration failure.

Source

Thrown at pkg/acquisition/modules/kafka/config.go:208

	if c.BatchConfiguration.BatchMaxBytes != 0 {
		rConf.MaxBytes = c.BatchConfiguration.BatchMaxBytes
	}

	if c.BatchConfiguration.BatchMaxWait != 0 {
		rConf.MaxWait = c.BatchConfiguration.BatchMaxWait
	}

	if c.BatchConfiguration.BatchQueueSize != 0 {
		rConf.QueueCapacity = c.BatchConfiguration.BatchQueueSize
	}

	if c.BatchConfiguration.CommitInterval != 0 {
		rConf.CommitInterval = c.BatchConfiguration.CommitInterval
	}

	if err := rConf.Validate(); err != nil {
		return &kafka.Reader{}, fmt.Errorf("while validating reader configuration: %w", err)
	}

	return kafka.NewReader(rConf), nil
}

View on GitHub (pinned to 909b515798)

Solutions

1. Read the wrapped %w Validate() message — it names the exact offending field
2. Ensure `batch.min_bytes` <= `batch.max_bytes` in the kafka acquisition yaml
3. Confirm `brokers:` resolves to a non-empty list (check templating/env substitution)
4. If unsure, temporarily drop batch tuning keys and re-add them one by one to isolate the invalid value

Example fix

// before
batch:
  min_bytes: 1048576
  max_bytes: 1024   # max < min -> Validate fails
// after
batch:
  min_bytes: 1024
  max_bytes: 1048576

if b := cfg.Batch; b.BatchMaxBytes != 0 && b.BatchMinBytes != 0 && b.BatchMaxBytes < b.BatchMinBytes {
    return fmt.Errorf("batch.max_bytes (%d) must be >= batch.min_bytes (%d)", b.BatchMaxBytes, b.BatchMinBytes)
}
if len(cfg.Brokers) == 0 {
    return errors.New("kafka: brokers list is empty")
}

if err := src.Configure(ctx, yamlCfg, logger, lvl); err != nil {
    if strings.Contains(err.Error(), "validating reader configuration") {
        logger.Errorf("kafka-go rejected the reader config: %v", err)
    }
    return err
}

When it happens

Trigger: Configure() (or a direct call to NewReader) with a Configuration whose derived kafka.ReaderConfig fails Validate() — e.g. empty Brokers slipped through, MaxBytes < MinBytes from batch settings, or invalid groupbalancer settings.

Common situations: Setting `batch.max_bytes` smaller than `batch.min_bytes` in the acquisition yaml; brokers list emptied by templating/variables that resolved to nothing; kafka-go version differences in Validate rules.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• while setting offset for reader on topic '%s': %w
• while closing %s reader on topic '%s': %w
• mtls is selected, but ca_cert is not provided
• invalid auth_type: must be one of basic_auth, headers, mtls
• server_cert is required

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/97a497b5e4b387f6. Report an issue: GitHub.