{"record":{"id":"75921f049e85677e","repo":"kubernetes/kops","slug":"error-removing-file-marker-versions-v","errorCode":null,"errorMessage":"error removing file/marker versions: %v","messagePattern":"error removing file/marker versions: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/s3fs.go","lineNumber":256,"sourceCode":"\t\t}\n\n\t\t// DeleteObjects can only process 1000 objects per call\n\t\tif len(objects) > 1000 {\n\t\t\trequest.Delete.Objects = objects[:1000]\n\t\t\tobjects = objects[1000:]\n\t\t} else {\n\t\t\trequest.Delete.Objects = objects\n\t\t\tobjects = nil\n\t\t}\n\n\t\tklog.V(8).Infof(\"removing %d file/marker versions\\n\", len(request.Delete.Objects))\n\n\t\tdeleteResult, err := client.DeleteObjects(ctx, request)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error removing %d file/marker versions: %v\", len(request.Delete.Objects), err)\n\t\t}\n\t\tif len(deleteResult.Errors) > 0 {\n\t\t\treturn fmt.Errorf(\"error removing file/marker versions: %v\", deleteResult.Errors)\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (p *S3Path) Join(relativePath ...string) Path {\n\targs := []string{p.key}\n\targs = append(args, relativePath...)\n\tjoined := path.Join(args...)\n\treturn &S3Path{\n\t\ts3Context: p.s3Context,\n\t\tbucket:    p.bucket,\n\t\tkey:       joined,\n\t\tscheme:    p.scheme,\n\t\tsse:       p.sse,\n\t\toptFn:     p.optFn,\n\t}","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/s3fs.go#L238-L274","documentation":"DeleteObjects can return HTTP 200 with per-object Errors inside the result. RemoveAllVersions checks deleteResult.Errors and throws \"error removing file/marker versions: [...]\" listing those per-key failures. This means the request succeeded but some individual version deletions were rejected.","triggerScenarios":"Calling RemoveAllVersions when specific version deletes fail server-side: permission denied on particular keys (key-level policy conditions), invalid version IDs (e.g. version removed concurrently), or protected objects (Object Lock/legal hold).","commonSituations":"Object Lock retention/legal hold on cluster state files; concurrent deletions of the same state store by two kOps runs; IAM key-prefix conditions that reject some keys in the batch.","solutions":["Inspect the listed per-object errors (Key, VersionId, Code, Message) in the error text","Remove Object Lock retention/legal hold or wait for the retention period to expire if objects are locked","Ensure no other process is deleting from the same state store concurrently (avoid parallel kOps deletes)","Adjust key-scoped IAM/bucket policy conditions that reject specific prefixes"],"exampleFix":"// before\nif len(deleteResult.Errors) > 0 { return fmt.Errorf(\"error removing file/marker versions: %v\", deleteResult.Errors) }\n// after (caller): surface key + code clearly\nvar failed []string\nfor _, e := range result.Errors { failed = append(failed, e.Key+\":\"+aws.ToString(e.Code)) }\nreturn fmt.Errorf(\"partial version delete failure: %s\", strings.Join(failed, \", \"))","handlingStrategy":"type-guard","validationCode":"// Detect Object Lock before attempting version purge\nlock, err := s3Client.GetObjectLockConfiguration(ctx, &s3.GetObjectLockConfigurationInput{Bucket: bucket})\nlocked := err == nil && lock.ObjectLockConfiguration.ObjectLockEnabled == types.ObjectLockEnabled","typeGuard":"func hasDeleteErrors(r *types.DeleteResult) bool { return r != nil && len(r.Errors) > 0 }","tryCatchPattern":"err := p.RemoveAllVersions(ctx)\nvar partials []types.Error\nif err != nil && strings.Contains(err.Error(), \"Code: AccessDenied\") {\n\t// inspect per-key failures referenced in err, skip locked objects\n}","preventionTips":["Check for S3 Object Lock/legal hold before purging versions","Serialize state-store deletions (no parallel kOps delete runs)","Log per-key Codes from partial delete failures for targeted follow-up"],"tags":["aws","s3","versioning","object-lock","partial-failure"],"backgroundTag":"s3-partial-delete-failure","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}