crowdsecurity/crowdsec · error

consumer_name is mandatory when use_enhanced_fanout is true

Error message

consumer_name is mandatory when use_enhanced_fanout is true

What it means

Enhanced fan-out requires a named, registered consumer on the stream. Configuration.Validate() rejects fan-out configs without consumer_name because the Kinesis SubscribeToShard API must be called against a specific consumer ARN derived from that name.

Source

Thrown at pkg/acquisition/modules/kinesis/config.go:85

		return err
	}

	s.Config = cfg

	return nil
}

func (c *Configuration) Validate() error {
	if c.StreamName == "" && !c.UseEnhancedFanOut {
		return errors.New("stream_name is mandatory when use_enhanced_fanout is false")
	}

	if c.StreamARN == "" && c.UseEnhancedFanOut {
		return errors.New("stream_arn is mandatory when use_enhanced_fanout is true")
	}

	if c.ConsumerName == "" && c.UseEnhancedFanOut {
		return errors.New("consumer_name is mandatory when use_enhanced_fanout is true")
	}

	if c.StreamARN != "" && c.StreamName != "" {
		return errors.New("stream_arn and stream_name are mutually exclusive")
	}

	return nil
}

func (s *Source) Configure(ctx context.Context, yamlConfig []byte, logger *log.Entry, metricsLevel metrics.AcquisitionMetricsLevel) error {
	s.logger = logger
	s.metricsLevel = metricsLevel

	err := s.UnmarshalConfig(yamlConfig)
	if err != nil {
		return err
	}

View on GitHub (pinned to 909b515798)

Solutions

1. Set consumer_name in the acquisition config (the consumer must be registered on the stream, e.g. via aws kinesis register-stream-consumer).
2. If a dedicated consumer is not wanted, set use_enhanced_fanout: false and identify the stream with stream_name instead.
3. Ensure the consumer name matches a registered consumer; the SDK does not create it automatically.

Example fix

// before (yaml)
source: kinesis
use_enhanced_fanout: true
stream_arn: arn:aws:kinesis:us-east-1:123456789012:stream/logs

// after (yaml)
source: kinesis
use_enhanced_fanout: true
stream_arn: arn:aws:kinesis:us-east-1:123456789012:stream/logs
consumer_name: crowdsec-consumer

if cfg.UseEnhancedFanOut && cfg.ConsumerName == "" {
    return fmt.Errorf("kinesis: consumer_name is required when use_enhanced_fanout is true")
}

When it happens

Trigger: Calling Validate() on a kinesis Configuration where UseEnhancedFanOut is true and ConsumerName is empty.

Common situations: Users enable use_enhanced_fanout: true and provide stream_arn but forget consumer_name, not realizing fan-out registers a dedicated consumer distinct from the default shared iterator.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• stream_arn is mandatory when use_enhanced_fanout is true
• stream_name is mandatory when use_enhanced_fanout is false
• stream_arn and stream_name are mutually exclusive
• cannot create kinesis client: %w
• failed to load aws config: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/0683222846b27f97. Report an issue: GitHub.