crowdsecurity/crowdsec · error

cannot specify both group_id and partition

Error message

cannot specify both group_id and partition

What it means

kafka-go readers work either as a consumer group (GroupID set, partitions auto-assigned) or pinned to a specific Partition (no group). NewReader (pkg/acquisition/modules/kafka/config.go:174) treats setting both as contradictory configuration and returns an empty reader plus this error.

Source

Thrown at pkg/acquisition/modules/kafka/config.go:174

		}

		dialer.TLS = tlsConfig
	}

	return dialer, nil
}

func (c *Configuration) NewReader(dialer *kafka.Dialer, logger *log.Entry) (*kafka.Reader, error) {
	rConf := kafka.ReaderConfig{
		Brokers:     c.Brokers,
		Topic:       c.Topic,
		Dialer:      dialer,
		Logger:      kafka.LoggerFunc(logger.Debugf),
		ErrorLogger: kafka.LoggerFunc(logger.Errorf),
	}

	if c.GroupID != "" && c.Partition != 0 {
		return &kafka.Reader{}, errors.New("cannot specify both group_id and partition")
	}

	if c.GroupID != "" {
		rConf.GroupID = c.GroupID
		// kafka-go does not support calling SetOffset while using a consumer group
		rConf.StartOffset = kafka.LastOffset
	} else if c.Partition != 0 {
		rConf.Partition = c.Partition
	} else {
		logger.Warnf("no group_id specified, crowdsec will only read from the 1st partition of the topic")
	}

	if c.BatchConfiguration.BatchMinBytes != 0 {
		rConf.MinBytes = c.BatchConfiguration.BatchMinBytes
	}

	if c.BatchConfiguration.BatchMaxBytes != 0 {
		rConf.MaxBytes = c.BatchConfiguration.BatchMaxBytes

View on GitHub (pinned to 909b515798)

Solutions

1. Decide on one mode: keep group_id and remove partition (group consumption, offsets managed by the group), or keep partition and remove group_id.
2. Remember partition: 0 is also treated as 'unset' — to use partition 0 with no group, omit group_id entirely.
3. Review the acquisition YAML so only one of the two keys is present.

Example fix

// before (acquis.yaml)
kafka:
  group_id: mygroup
  partition: 3

// after (group mode)
kafka:
  group_id: mygroup
// or (manual partition mode)
kafka:
  partition: 3

if cfg.GroupID != "" && cfg.Partition != 0 {
    return errors.New("kafka source: set group_id OR partition, not both")
}

func validKafkaConsumerMode(cfg struct{ GroupID string; Partition int }) bool {
    return !(cfg.GroupID != "" && cfg.Partition != 0)
}

r, err := kafka.NewReader(conf)
if err != nil && strings.Contains(err.Error(), "cannot specify both group_id and partition") {
    // drop one of the two settings and rebuild the reader
}

When it happens

Trigger: Creating a kafka source/reader whose Configuration has both a non-empty group_id and a non-zero partition, via YAML config or by calling NewReader directly.

Common situations: Merging a consumer-group template config with a manual-partition example; leftover group_id from an earlier config while partition was added for offset control; copy-paste combining two acquisition snippets.

Related errors

• path must start with /
• basic_auth is selected, but basic_auth is not provided
• basic_auth is selected, but username is not provided
• basic_auth is selected, but password is not provided
• headers is selected, but headers is not provided

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/88505d796109b7f0. Report an issue: GitHub.