apache/beam · error

error deleting object

Error message

error deleting object %s: %v

What it means

Returned by fs.Remove when the AWS SDK DeleteObject call fails. The error wraps the underlying AWS error with the filename. Common causes include missing delete permissions, a non-existent bucket, or network/API failures.

Solutions

1. Grant the caller IAM permission s3:DeleteObject on the bucket/key prefix.
2. Verify the bucket exists and matches the configured region/credentials.
3. Check for S3 Object Lock, retention policies, or bucket policies denying deletion.
4. Retry on transient errors; inspect the wrapped cause for the exact API error code.

Example fix

// before: single attempt, no retry
if _, err = f.client.DeleteObject(ctx, params); err != nil {
    return fmt.Errorf("error deleting object %s: %v", filename, err)
}

// after: distinguish not-found and retry transient errors
if _, err = f.client.DeleteObject(ctx, params); err != nil {
    var nfe *types.NoSuchBucket
    if errors.As(err, &nfe) { return nil }
    if aerr, ok := err.(awserr.Error); ok && aerr.RetryableError() != nil { /* backoff + retry */ }
    return fmt.Errorf("error deleting object %s: %w", filename, err)
}

if !strings.HasPrefix(filename, "s3://") { return fmt.Errorf("not an s3 URI: %s", filename) }
// confirm delete permission is configured for the job's IAM role out-of-band

err := fsys.Remove(ctx, filename)
if err != nil {
    if isTransientAWSError(err) { /* backoff and retry Remove */ }
    if strings.Contains(err.Error(), "AccessDenied") { return ErrPermissionDenied }
    return err
}

When it happens

Trigger: Calling fs.Remove(ctx, filename) on a valid s3:// path where the IAM role lacks s3:DeleteObject, the bucket is in another account/region, or the S3 service returns an error.

Common situations: Read-only credentials used by a cleanup job; bucket name typo; bucket versioning/retention policies blocking deletion; transient network failures.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• error copying object
• error getting metadata for object
• error getting metadata for object
• error getting object
• error listing object keys

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9f02d848b6cf2d4b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:202

		return time.Time{}, fmt.Errorf("error getting metadata for object %s: %v", filename, err)
	}

	return aws.ToTime(output.LastModified), err
}

// Remove removes the file from the filesystem.
func (f *fs) Remove(ctx context.Context, filename string) error {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.DeleteObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	if _, err = f.client.DeleteObject(ctx, params); err != nil {
		return fmt.Errorf("error deleting object %s: %v", filename, err)
	}

	return nil
}

// Copy copies the file from the old path to the new path.
func (f *fs) Copy(ctx context.Context, oldpath, newpath string) error {
	sourceBucket, sourceKey, err := parseURI(oldpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 source uri %s: %v", oldpath, err)
	}

	copySource := fmt.Sprintf("%s/%s", sourceBucket, sourceKey)
	destBucket, destKey, err := parseURI(newpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 destination uri %s: %v", newpath, err)
	}

View on GitHub (pinned to 12126d8942)