{"record":{"id":"cd288fb3dfa70947","repo":"kubernetes/kops","slug":"error-removing-d-files-w","errorCode":null,"errorMessage":"error removing %d files: %w","messagePattern":"error removing (.+?) files: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/s3fs.go","lineNumber":172,"sourceCode":"\n\trequest := &s3.DeleteObjectsInput{\n\t\tBucket: aws.String(p.bucket),\n\t\tDelete: &types.Delete{},\n\t}\n\n\tfor len(objects) > 0 {\n\t\t// DeleteObjects can only process 1000 objects per call\n\t\tif len(objects) > 1000 {\n\t\t\trequest.Delete.Objects = objects[:1000]\n\t\t\tobjects = objects[1000:]\n\t\t} else {\n\t\t\trequest.Delete.Objects = objects\n\t\t\tobjects = nil\n\t\t}\n\n\t\t_, err = client.DeleteObjects(ctx, request)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error removing %d files: %w\", len(request.Delete.Objects), err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (p *S3Path) RemoveAllVersions(ctx context.Context) error {\n\tclient, err := p.client(ctx)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tklog.V(8).Infof(\"removing all versions of file %s\", p)\n\n\trequest := &s3.ListObjectVersionsInput{\n\t\tBucket: aws.String(p.bucket),\n\t\tPrefix: aws.String(p.key),\n\t}","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/s3fs.go#L154-L190","documentation":"RemoveAll deletes tree objects in batches of up to 1000 via the DeleteObjects (multi-delete) API. If the whole batched DeleteObjects call fails at the transport/API level, the error is wrapped as \"error removing N files\" where N is the batch size.","triggerScenarios":"Calling RemoveAll on an S3Path when DeleteObjects fails: IAM policy grants DeleteObject but not DeleteObjects on the bucket; more than 1000 keys per request (impossible — batches are chunked, so this is API-level failure); throttling or network failure during the batch call.","commonSituations":"IAM policies missing s3:DeleteObjects (multi-object delete) while allowing single-object deletes; large cluster deletions triggering S3 throttling (SlowDown); VPC endpoint policy blocking the DeleteObjects action.","solutions":["Check the wrapped AWS error: AccessDenied → add s3:DeleteObjects to the state-store IAM policy","Retry the operation on transient errors (SlowDown/RequestTimeout); batches already nil out after send","Verify VPC endpoint / SCP policies allow s3:DeleteObjects","If S3 versioning is enabled, consider RemoveAllVersions instead to purge all versions and markers"],"exampleFix":"// before\n_, err = client.DeleteObjects(ctx, request)\nif err != nil { return fmt.Errorf(\"error removing %d files: %w\", len(request.Delete.Objects), err) }\n// after (caller): retry transient batch failures\nerr = wait.ExponentialBackoff(backoff, func() (bool, error) {\n\t_, err := retryRemoveAll(ctx)\n\treturn err == nil, nil\n})","handlingStrategy":"retry","validationCode":"// Verify multi-delete permission with a dry-run style call on a scratch key\n_, err := s3Client.DeleteObjects(ctx, &s3.DeleteObjectsInput{\n\tBucket: bucket,\n\tDelete: &types.Delete{Objects: []types.ObjectIdentifier{{Key: aws.String(\"__perm_test__\")}}},\n})","typeGuard":null,"tryCatchPattern":"err := wait.ExponentialBackoff(retryBackoff, func() (bool, error) {\n\terr := root.RemoveAll(ctx)\n\tif err != nil && (AWSErrorCode(err) == \"SlowDown\" || AWSErrorCode(err) == \"InternalError\") {\n\t\treturn false, nil\n\t}\n\treturn err == nil, err\n})","preventionTips":["Include s3:DeleteObjects in state-store IAM policies","Avoid concurrent deletions against the same bucket prefix","Use RemoveAllVersions for versioned buckets so nothing is left behind"],"tags":["aws","s3","batch-delete","iam"],"backgroundTag":"s3-batch-delete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}