{"record":{"id":"c75972ef1ca1babd","repo":"crowdsecurity/crowdsec","slug":"cannot-get-records-w","errorCode":null,"errorMessage":"cannot get records: %w","messagePattern":"cannot get records: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/kinesis/run.go","lineNumber":369,"sourceCode":"\t\tcase <-ticker.C:\n\t\t\trecords, err := s.kClient.GetRecords(ctx, &kinesis.GetRecordsInput{ShardIterator: it})\n\n\t\t\tvar throughputErr *kinTypes.ProvisionedThroughputExceededException\n\t\t\tif errors.As(err, &throughputErr) {\n\t\t\t\tlogger.Warn(\"Provisioned throughput exceeded\")\n\t\t\t\t// TODO: implement exponential backoff\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tvar expiredIteratorErr *kinTypes.ExpiredIteratorException\n\t\t\tif errors.As(err, &expiredIteratorErr) {\n\t\t\t\tlogger.Warn(\"Expired iterator\")\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif err != nil {\n\t\t\t\tlogger.Error(\"Cannot get records\")\n\t\t\t\treturn fmt.Errorf(\"cannot get records: %w\", err)\n\t\t\t}\n\n\t\t\tit = records.NextShardIterator\n\n\t\t\ts.ParseAndPushRecords(records.Records, out, logger, shardID)\n\n\t\t\tif it == nil {\n\t\t\t\tlogger.Warnf(\"Shard has been closed\")\n\t\t\t\treturn nil\n\t\t\t}\n\t\tcase <-s.shardReaderTomb.Dying():\n\t\t\tlogger.Infof(\"shardReaderTomb is dying, exiting ReadFromShard\")\n\t\t\tticker.Stop()\n\n\t\t\treturn nil\n\t\t}\n\t}\n}","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/kinesis/run.go#L351-L387","documentation":"GetRecords failed with an error other than ProvisionedThroughputExceeded or ExpiredIterator (those two are handled above it and just continue). Any other AWS error — permissions, invalid iterator, internal service error, networking — terminates the shard reader goroutine. This is the steady-state read path of classic polling mode.","triggerScenarios":"The ticker fires and s.kClient.GetRecords is called with the current shard iterator; the call returns e.g. AccessDeniedException (missing kinesis:GetRecords), InvalidArgumentException (iterator invalidated by a KMS key change on an encrypted stream), KMSThrottlingException, or a 5xx/network error not matching the two handled exception types.","commonSituations":"KMS key rotated/disabled on an encrypted stream invalidating iterators; IAM changes mid-run removing GetRecords; sustained AWS-side errors; TLS/proxy failures cutting the connection.","solutions":["Check IAM for kinesis:GetRecords (and KMS Decrypt if the stream is encrypted).","Add errors.As handling for KMSThrottlingException/InvalidArgumentException analogous to the existing throughput/expired cases, with retry/backoff.","If iterators are being invalidated (KMS), restart the datasource so GetShardIterator issues a fresh iterator.","Verify network path to kinesis.<region>.amazonaws.com; retry the goroutine via the outer ReadFromStream loop."],"exampleFix":"// before: any other error kills the reader\nif err != nil {\n\treturn fmt.Errorf(\"cannot get records: %w\", err)\n}\n// after: tolerate KMS throttling like throughput errors\nvar kmsErr *kinTypes.KMSThrottlingException\nif errors.As(err, &kmsErr) {\n\tlogger.Warn(\"KMS throttling, backing off\")\n\tcontinue\n}\nreturn fmt.Errorf(\"cannot get records: %w\", err)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Mirror the existing handling for more exception types:\nvar kmsThr *kinTypes.KMSThrottlingException\nvar invalidArg *kinTypes.InvalidArgumentException\nswitch {\ncase errors.As(err, &kmsThr):\n\tlogger.Warn(\"KMS throttling\"); continue\ncase errors.As(err, &invalidArg):\n\t// iterator invalid: re-acquire via GetShardIterator\n\tlogger.Warn(\"iterator invalid, re-acquiring\"); continue\ndefault:\n\treturn fmt.Errorf(\"cannot get records: %w\", err)\n}","preventionTips":["If the stream is KMS-encrypted, monitor key state and grant decrypt to the datasource role.","Errors.As on typed AWS exceptions before treating an error as fatal.","Keep retry/backoff for 5xx and throttling rather than killing the reader goroutine.","Alert on repeated 'Cannot get records' to catch IAM regressions early."],"tags":["aws","kinesis","streaming","network"],"backgroundTag":"api-request-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}