crowdsecurity/crowdsec · error

cannot get records: %w

Error message

cannot get records: %w

What it means

GetRecords failed with an error other than ProvisionedThroughputExceeded or ExpiredIterator (those two are handled above it and just continue). Any other AWS error — permissions, invalid iterator, internal service error, networking — terminates the shard reader goroutine. This is the steady-state read path of classic polling mode.

Source

Thrown at pkg/acquisition/modules/kinesis/run.go:369

		case <-ticker.C:
			records, err := s.kClient.GetRecords(ctx, &kinesis.GetRecordsInput{ShardIterator: it})

			var throughputErr *kinTypes.ProvisionedThroughputExceededException
			if errors.As(err, &throughputErr) {
				logger.Warn("Provisioned throughput exceeded")
				// TODO: implement exponential backoff
				continue
			}

			var expiredIteratorErr *kinTypes.ExpiredIteratorException
			if errors.As(err, &expiredIteratorErr) {
				logger.Warn("Expired iterator")
				continue
			}

			if err != nil {
				logger.Error("Cannot get records")
				return fmt.Errorf("cannot get records: %w", err)
			}

			it = records.NextShardIterator

			s.ParseAndPushRecords(records.Records, out, logger, shardID)

			if it == nil {
				logger.Warnf("Shard has been closed")
				return nil
			}
		case <-s.shardReaderTomb.Dying():
			logger.Infof("shardReaderTomb is dying, exiting ReadFromShard")
			ticker.Stop()

			return nil
		}
	}
}

View on GitHub (pinned to 909b515798)

Solutions

1. Check IAM for kinesis:GetRecords (and KMS Decrypt if the stream is encrypted).
2. Add errors.As handling for KMSThrottlingException/InvalidArgumentException analogous to the existing throughput/expired cases, with retry/backoff.
3. If iterators are being invalidated (KMS), restart the datasource so GetShardIterator issues a fresh iterator.
4. Verify network path to kinesis.<region>.amazonaws.com; retry the goroutine via the outer ReadFromStream loop.

Example fix

// before: any other error kills the reader
if err != nil {
	return fmt.Errorf("cannot get records: %w", err)
}
// after: tolerate KMS throttling like throughput errors
var kmsErr *kinTypes.KMSThrottlingException
if errors.As(err, &kmsErr) {
	logger.Warn("KMS throttling, backing off")
	continue
}
return fmt.Errorf("cannot get records: %w", err)

// Mirror the existing handling for more exception types:
var kmsThr *kinTypes.KMSThrottlingException
var invalidArg *kinTypes.InvalidArgumentException
switch {
case errors.As(err, &kmsThr):
	logger.Warn("KMS throttling"); continue
case errors.As(err, &invalidArg):
	// iterator invalid: re-acquire via GetShardIterator
	logger.Warn("iterator invalid, re-acquiring"); continue
default:
	return fmt.Errorf("cannot get records: %w", err)
}

When it happens

Trigger: The ticker fires and s.kClient.GetRecords is called with the current shard iterator; the call returns e.g. AccessDeniedException (missing kinesis:GetRecords), InvalidArgumentException (iterator invalidated by a KMS key change on an encrypted stream), KMSThrottlingException, or a 5xx/network error not matching the two handled exception types.

Common situations: KMS key rotated/disabled on an encrypted stream invalidating iterators; IAM changes mid-run removing GetRecords; sustained AWS-side errors; TLS/proxy failures cutting the connection.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• while reading %s/%s: %w
• cannot list shards for enhanced_read: %w
• cannot subscribe to shard: %w
• cannot subscribe to shards: %w
• cannot get shard iterator: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/c75972ef1ca1babd. Report an issue: GitHub.