{"record":{"id":"f5df4cfc13787463","repo":"kubernetes/kops","slug":"error-removing-d-file-marker-versions-v","errorCode":null,"errorMessage":"error removing %d file/marker versions: %v","messagePattern":"error removing (.+?) file/marker versions: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/s3fs.go","lineNumber":253,"sourceCode":"\t\trequest := &s3.DeleteObjectsInput{\n\t\t\tBucket: aws.String(p.bucket),\n\t\t\tDelete: &types.Delete{},\n\t\t}\n\n\t\t// DeleteObjects can only process 1000 objects per call\n\t\tif len(objects) > 1000 {\n\t\t\trequest.Delete.Objects = objects[:1000]\n\t\t\tobjects = objects[1000:]\n\t\t} else {\n\t\t\trequest.Delete.Objects = objects\n\t\t\tobjects = nil\n\t\t}\n\n\t\tklog.V(8).Infof(\"removing %d file/marker versions\\n\", len(request.Delete.Objects))\n\n\t\tdeleteResult, err := client.DeleteObjects(ctx, request)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error removing %d file/marker versions: %v\", len(request.Delete.Objects), err)\n\t\t}\n\t\tif len(deleteResult.Errors) > 0 {\n\t\t\treturn fmt.Errorf(\"error removing file/marker versions: %v\", deleteResult.Errors)\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (p *S3Path) Join(relativePath ...string) Path {\n\targs := []string{p.key}\n\targs = append(args, relativePath...)\n\tjoined := path.Join(args...)\n\treturn &S3Path{\n\t\ts3Context: p.s3Context,\n\t\tbucket:    p.bucket,\n\t\tkey:       joined,\n\t\tscheme:    p.scheme,","sourceCodeStart":235,"sourceCodeEnd":271,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/s3fs.go#L235-L271","documentation":"After collecting all versions and delete markers, RemoveAllVersions removes them in batched DeleteObjects calls. If the call itself fails, this error reports how many file/marker versions were in the failed request batch.","triggerScenarios":"Calling RemoveAllVersions on a versioned bucket when DeleteObjects (with VersionId set per object) fails: IAM missing s3:DeleteObjectVersion; throttling; >1000 versions chunked correctly but a chunk fails on network/API error.","commonSituations":"Roles allowed to delete current objects but not specific old versions (s3:DeleteObjectVersion absent); MFA-delete enabled buckets; transient SlowDown during large multi-version cleanups.","solutions":["Add s3:DeleteObjectVersion to the IAM policy for the state-store bucket","Disable MFA-delete or use MFA-authenticated deletion if MFA-delete is on","Retry the operation on throttling/transient errors","Verify no bucket policy/SCP denies DeleteObjectVersion for the principal"],"exampleFix":"// before\n_, err := client.DeleteObjects(ctx, request)\nif err != nil { return fmt.Errorf(\"error removing %d file/marker versions: %v\", len(request.Delete.Objects), err) }\n// after (caller): retry with backoff\nbackoff := wait.Backoff{Steps: 5, Duration: time.Second, Factor: 2}\nerr = wait.ExponentialBackoff(backoff, func() (bool, error) { return retryRemoveAllVersions(ctx) == nil, nil })","handlingStrategy":"retry","validationCode":"// Ensure version deletion is permitted before bulk cleanup\n_, err := s3Client.DeleteObjectVersion(ctx, &s3.DeleteObjectVersionInput{Bucket: bucket, Key: k, VersionId: v})\n// AccessDenied ⇒ s3:DeleteObjectVersion missing","typeGuard":null,"tryCatchPattern":"if err := p.RemoveAllVersions(ctx); err != nil {\n\tif AWSErrorCode(err) == \"AccessDenied\" { return errMissingDeleteObjectVersion }\n\t// else retry transient codes\n}","preventionTips":["Grant s3:DeleteObjectVersion for versioned state stores","Avoid MFA-delete on kOps state buckets","Stagger large cleanups to avoid S3 SlowDown throttling"],"tags":["aws","s3","versioning","batch-delete","iam"],"backgroundTag":"s3-version-delete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}