crowdsecurity/crowdsec · error

stream_arn is mandatory when use_enhanced_fanout is true

Error message

stream_arn is mandatory when use_enhanced_fanout is true

What it means

In enhanced fan-out mode the Kinesis source registers a dedicated consumer identified by the stream ARN, so StreamARN must be set. Configuration.Validate() enforces that any config with use_enhanced_fanout: true also provides stream_arn. Stream name alone is not accepted because the fan-out subscribe API requires the full ARN.

Source

Thrown at pkg/acquisition/modules/kinesis/config.go:81

func (s *Source) UnmarshalConfig(yamlConfig []byte) error {
	cfg, err := ConfigurationFromYAML(yamlConfig)
	if err != nil {
		return err
	}

	s.Config = cfg

	return nil
}

func (c *Configuration) Validate() error {
	if c.StreamName == "" && !c.UseEnhancedFanOut {
		return errors.New("stream_name is mandatory when use_enhanced_fanout is false")
	}

	if c.StreamARN == "" && c.UseEnhancedFanOut {
		return errors.New("stream_arn is mandatory when use_enhanced_fanout is true")
	}

	if c.ConsumerName == "" && c.UseEnhancedFanOut {
		return errors.New("consumer_name is mandatory when use_enhanced_fanout is true")
	}

	if c.StreamARN != "" && c.StreamName != "" {
		return errors.New("stream_arn and stream_name are mutually exclusive")
	}

	return nil
}

func (s *Source) Configure(ctx context.Context, yamlConfig []byte, logger *log.Entry, metricsLevel metrics.AcquisitionMetricsLevel) error {
	s.logger = logger
	s.metricsLevel = metricsLevel

	err := s.UnmarshalConfig(yamlConfig)

View on GitHub (pinned to 909b515798)

Solutions

1. Set stream_arn to the full stream ARN (arn:aws:kinesis:<region>:<account>:stream/<name>) in the acquisition config.
2. Keep use_enhanced_fanout: true only if you have registered a consumer; otherwise drop the flag and use stream_name.
3. Also set consumer_name, which is required in the same mode.

Example fix

// before (yaml)
source: kinesis
use_enhanced_fanout: true
stream_name: logs
consumer_name: crowdsec

// after (yaml)
source: kinesis
use_enhanced_fanout: true
stream_arn: arn:aws:kinesis:us-east-1:123456789012:stream/logs
consumer_name: crowdsec

if cfg.UseEnhancedFanOut && cfg.StreamARN == "" {
    return fmt.Errorf("kinesis: stream_arn is required when use_enhanced_fanout is true")
}

When it happens

Trigger: Calling Validate() on a kinesis Configuration where UseEnhancedFanOut is true and StreamARN is empty — e.g. a config with use_enhanced_fanout: true and only stream_name set.

Common situations: Users switch on enhanced fan-out but keep using stream_name (the shared-consumer style config), or copy a fan-out config template and leave stream_arn blank/placeholder.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• consumer_name is mandatory when use_enhanced_fanout is true
• stream_name is mandatory when use_enhanced_fanout is false
• stream_arn and stream_name are mutually exclusive
• cannot create kinesis client: %w
• failed to load aws config: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/ffffed819aea7903. Report an issue: GitHub.