crowdsecurity/crowdsec · error

cannot create kinesis client: %w

Error message

cannot create kinesis client: %w

What it means

Wraps any failure from Source.newClient while configuring the kinesis datasource. newClient loads the AWS SDK config (config.LoadDefaultConfig) and constructs a kinesis client; it fails when the AWS shared config/credentials cannot be loaded (bad profile, unreadable ~/.aws files, malformed env vars). The kinesis client itself is built unconditionally, so nearly all wrapped errors come from AWS config loading.

Source

Thrown at pkg/acquisition/modules/kinesis/config.go:106

	if c.StreamARN != "" && c.StreamName != "" {
		return errors.New("stream_arn and stream_name are mutually exclusive")
	}

	return nil
}

func (s *Source) Configure(ctx context.Context, yamlConfig []byte, logger *log.Entry, metricsLevel metrics.AcquisitionMetricsLevel) error {
	s.logger = logger
	s.metricsLevel = metricsLevel

	err := s.UnmarshalConfig(yamlConfig)
	if err != nil {
		return err
	}

	err = s.newClient(ctx)
	if err != nil {
		return fmt.Errorf("cannot create kinesis client: %w", err)
	}

	s.shardReaderTomb = &tomb.Tomb{}

	return nil
}

func (s *Source) newClient(ctx context.Context) error {
	var loadOpts []func(*config.LoadOptions) error
	if s.Config.AwsProfile != nil && *s.Config.AwsProfile != "" {
		loadOpts = append(loadOpts, config.WithSharedConfigProfile(*s.Config.AwsProfile))
	}

	region := s.Config.AwsRegion
	if region == "" {
		region = "us-east-1"
	}

View on GitHub (pinned to 909b515798)

Solutions

1. Read the wrapped error — 'failed to load aws config: %w' names the profile or file that failed
2. Verify the profile exists: `aws configure list-profiles` and that ~/.aws/config & ~/.aws/credentials are readable by the crowdsec process
3. If using env credentials, check AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_REGION are valid and well-formed
4. Remove `aws_profile:` if you intend to use env/instance credentials instead
5. Ensure aws_endpoint (if set) is a valid URL for localstack-style testing

Example fix

// before
source: kinesis
aws_profile: prod-old   # profile not in ~/.aws/config
// after
source: kinesis
aws_profile: default

// before startup
if profile := cfg.AwsProfile; profile != nil && *profile != "" {
    out, err := exec.Command("aws", "configure", "list-profiles").Output()
    if err != nil || !strings.Contains(string(out), *profile) {
        return fmt.Errorf("aws profile %q not found in shared config", *profile)
    }
}

if err := src.Configure(ctx, yamlCfg, logger, lvl); err != nil {
    if strings.Contains(err.Error(), "failed to load aws config") {
        logger.Errorf("check aws_profile/shared credentials files: %v", err)
    }
    return fmt.Errorf("kinesis datasource setup failed: %w", err)
}

When it happens

Trigger: Configure() -> s.newClient(ctx): config.LoadDefaultConfig fails due to a specified `aws_profile` missing from ~/.aws/config or credentials, malformed AWS_* env variables, or unreadable shared config files. Note the SDK defers credential errors — LoadDefaultConfig mostly fails on config-file/profile problems.

Common situations: aws_profile references a profile that doesn't exist; AWS_CONFIG_FILE/AWS_SHARED_CREDENTIALS_FILE pointing to missing files; malformed values for AWS_RETRY_MODE etc.; container lacking the mounted aws config directory.

Related errors

• failed to load aws config: %w
• cannot parse: %s
• path must start with /
• stream_name is mandatory when use_enhanced_fanout is false
• stream_arn is mandatory when use_enhanced_fanout is true

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/87bff5630ddcca48. Report an issue: GitHub.