{"record":{"id":"d7f443c7784862bd","repo":"vitessio/vitess","slug":"objerror-message","errorCode":null,"errorMessage":"${objError.Message}","messagePattern":"\\$\\{objError\\.Message\\}","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/mysqlctl/s3backupstorage/s3.go","lineNumber":526,"sourceCode":"\t\t\tobjIds = append(objIds, types.ObjectIdentifier{\n\t\t\t\tKey: obj.Key,\n\t\t\t})\n\t\t}\n\n\t\tquiet := true // return less in the Delete response\n\t\tout, err := c.DeleteObjects(ctx, &s3.DeleteObjectsInput{\n\t\t\tBucket: &bucket,\n\t\t\tDelete: &types.Delete{\n\t\t\t\tObjects: objIds,\n\t\t\t\tQuiet:   &quiet,\n\t\t\t},\n\t\t})\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\tfor _, objError := range out.Errors {\n\t\t\treturn errors.New(*objError.Message)\n\t\t}\n\n\t\tif objs.NextContinuationToken == nil {\n\t\t\tbreak\n\t\t}\n\n\t\tquery.ContinuationToken = objs.NextContinuationToken\n\t}\n\n\treturn nil\n}\n\n// Close is part of the backupstorage.BackupStorage interface.\nfunc (bs *S3BackupStorage) Close() error {\n\tbs.mu.Lock()\n\tdefer bs.mu.Unlock()\n\tbs._client = nil\n\tbs.s3SSE.reset()","sourceCodeStart":508,"sourceCodeEnd":544,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/mysqlctl/s3backupstorage/s3.go#L508-L544","documentation":"RemoveBackup lists and batch-deletes (DeleteObjects) all objects under a backup prefix. The S3 API can succeed per-request while still returning per-object failures in out.Errors; this code surfaces the first object error message verbatim, so the message text comes from AWS, not Vitess.","triggerScenarios":"Running RemoveBackup against a bucket where some objects cannot be deleted — e.g. objects locked by S3 Object Lock, permission denied on specific keys, or malformed keys returned by the listing.","commonSituations":"Buckets with Object Lock/retention (Compliance mode), IAM policies restricting DeleteObject on a prefix, versioned buckets with delete markers interacting with lifecycle rules, partially-visible objects due to scoped credentials.","solutions":["Read objError.Message/Key/Code in the log — the AWS message identifies the failing key and reason","Grant the credentials s3:DeleteObject on all keys under the backup prefix","Check for S3 Object Lock or retention configuration blocking deletes on affected objects","Collect and aggregate all out.Errors instead of returning on the first to know full scope"],"exampleFix":"// before\nfor _, objError := range out.Errors {\n    return errors.New(*objError.Message)\n}\n// after\nvar errs []string\nfor _, objError := range out.Errors {\n    errs = append(errs, fmt.Sprintf(\"%s: %s\", *objError.Key, *objError.Message))\n}\nif len(errs) > 0 {\n    return errors.New(strings.Join(errs, \"; \"))\n}","handlingStrategy":"try-catch","validationCode":"// pre-check permissions before RemoveBackup\n_, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: &bucket, Key: &probeKey})\nif err != nil {\n    return fmt.Errorf(\"no access to bucket objects: %w\", err)\n}","typeGuard":"func hasDeleteErrors(out *s3.DeleteObjectsOutput) bool {\n    return out != nil && len(out.Errors) > 0\n}","tryCatchPattern":"err := bs.RemoveBackup(ctx, dir, name)\nif err != nil {\n    var ae smithy.APIError\n    if errors.As(err, &ae) {\n        log.Warnf(\"S3 delete failed: code=%s msg=%s\", ae.ErrorCode(), ae.ErrorMessage())\n    }\n    // inspect object lock / IAM and retry after remediation\n}","preventionTips":["Confirm credentials have s3:DeleteObject on the backup prefix","Check for S3 Object Lock/retention before choosing the bucket for backups","Retry RemoveBackup: partial deletions are common; a second pass clears leftovers","Log all DeleteObjects errors (Key, Code, Message), not just the first"],"tags":["s3","aws","delete","permissions"],"backgroundTag":"s3-delete-object-error","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}