vitessio/vitess · error

${objError.Message}

Error message

${objError.Message}

What it means

RemoveBackup lists and batch-deletes (DeleteObjects) all objects under a backup prefix. The S3 API can succeed per-request while still returning per-object failures in out.Errors; this code surfaces the first object error message verbatim, so the message text comes from AWS, not Vitess.

Source

Thrown at go/vt/mysqlctl/s3backupstorage/s3.go:526

			objIds = append(objIds, types.ObjectIdentifier{
				Key: obj.Key,
			})
		}

		quiet := true // return less in the Delete response
		out, err := c.DeleteObjects(ctx, &s3.DeleteObjectsInput{
			Bucket: &bucket,
			Delete: &types.Delete{
				Objects: objIds,
				Quiet:   &quiet,
			},
		})
		if err != nil {
			return err
		}

		for _, objError := range out.Errors {
			return errors.New(*objError.Message)
		}

		if objs.NextContinuationToken == nil {
			break
		}

		query.ContinuationToken = objs.NextContinuationToken
	}

	return nil
}

// Close is part of the backupstorage.BackupStorage interface.
func (bs *S3BackupStorage) Close() error {
	bs.mu.Lock()
	defer bs.mu.Unlock()
	bs._client = nil
	bs.s3SSE.reset()

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read objError.Message/Key/Code in the log — the AWS message identifies the failing key and reason
2. Grant the credentials s3:DeleteObject on all keys under the backup prefix
3. Check for S3 Object Lock or retention configuration blocking deletes on affected objects
4. Collect and aggregate all out.Errors instead of returning on the first to know full scope

Example fix

// before
for _, objError := range out.Errors {
    return errors.New(*objError.Message)
}
// after
var errs []string
for _, objError := range out.Errors {
    errs = append(errs, fmt.Sprintf("%s: %s", *objError.Key, *objError.Message))
}
if len(errs) > 0 {
    return errors.New(strings.Join(errs, "; "))
}

// pre-check permissions before RemoveBackup
_, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: &bucket, Key: &probeKey})
if err != nil {
    return fmt.Errorf("no access to bucket objects: %w", err)
}

func hasDeleteErrors(out *s3.DeleteObjectsOutput) bool {
    return out != nil && len(out.Errors) > 0
}

err := bs.RemoveBackup(ctx, dir, name)
if err != nil {
    var ae smithy.APIError
    if errors.As(err, &ae) {
        log.Warnf("S3 delete failed: code=%s msg=%s", ae.ErrorCode(), ae.ErrorMessage())
    }
    // inspect object lock / IAM and retry after remediation
}

When it happens

Trigger: Running RemoveBackup against a bucket where some objects cannot be deleted — e.g. objects locked by S3 Object Lock, permission denied on specific keys, or malformed keys returned by the listing.

Common situations: Buckets with Object Lock/retention (Compliance mode), IAM policies restricting DeleteObject on a prefix, versioned buckets with delete markers interacting with lifecycle rules, partially-visible objects due to scoped credentials.

Related errors

• ErrPartSize
• AddFile cannot be called on read-only backup
• EndBackup cannot be called on read-only backup
• AbortBackup cannot be called on read-only backup
• ReadFile cannot be called on read-write backup

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d7f443c7784862bd. Report an issue: GitHub.