apache/beam · error

error copying object

Error message

error copying object %s: %v

What it means

Returned by fs.Copy when the AWS SDK CopyObject call fails. The message reports the source path while wrapping the underlying AWS error. Causes include missing s3:GetObject on the source or s3:PutObject on the destination, missing CopySource URI-encoding, or size limits (single CopyObject fails for objects >5 GB).

Solutions

1. Ensure IAM allows s3:GetObject on source and s3:PutObject (plus s3:GetObject for ACL copying) on destination.
2. Verify the source object exists and that the key is URI-encoded when forming CopySource.
3. For objects over 5 GB, use multipart upload/copy instead of single CopyObject.
4. Inspect the wrapped AWS error code and retry transient failures.

Example fix

// before: unencoded CopySource
copySource := fmt.Sprintf("%s/%s", sourceBucket, sourceKey)

// after: URL-encode the key per S3 CopySource requirements
copySource := fmt.Sprintf("%s/%s", sourceBucket, url.PathEscape(sourceKey))

validS3URI(oldpath); validS3URI(newpath)
// for keys with spaces/special chars, verify encoding:
if strings.ContainsAny(sourceKey, " ") { copySource = sourceBucket + "/" + url.PathEscape(sourceKey) }

if _, err = f.client.CopyObject(ctx, params); err != nil {
    if isTransientAWSError(err) { /* backoff + retry */ }
    switch {
    case strings.Contains(err.Error(), "AccessDenied"):
        return ErrPermission
    case strings.Contains(err.Error(), "NoSuchKey"):
        return ErrSourceMissing
    case strings.Contains(err.Error(), "InvalidRequest"):
        return ErrObjectTooLarge // >5GB needs multipart copy
    }
    return err
}

When it happens

Trigger: Calling fs.Copy on valid URIs where IAM denies the copy, the source key does not exist, CopySource is not URL-encoded (keys with spaces/special chars), or the object exceeds 5 GB.

Common situations: Cross-account copies without proper bucket policy grants; keys with special characters not encoded; large files needing multipart copy; wrong region endpoints.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• error deleting object
• error getting metadata for object
• error getting metadata for object
• error getting object
• error listing object keys

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3cc0a5c42b7be881. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:227

func (f *fs) Copy(ctx context.Context, oldpath, newpath string) error {
	sourceBucket, sourceKey, err := parseURI(oldpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 source uri %s: %v", oldpath, err)
	}

	copySource := fmt.Sprintf("%s/%s", sourceBucket, sourceKey)
	destBucket, destKey, err := parseURI(newpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 destination uri %s: %v", newpath, err)
	}

	params := &s3.CopyObjectInput{
		Bucket:     aws.String(destBucket),
		CopySource: aws.String(copySource),
		Key:        aws.String(destKey),
	}
	if _, err = f.client.CopyObject(ctx, params); err != nil {
		return fmt.Errorf("error copying object %s: %v", oldpath, err)
	}

	return nil
}

// Compile time check for interface implementations.
var (
	_ filesystem.LastModifiedGetter = (*fs)(nil)
	_ filesystem.Remover            = (*fs)(nil)
	_ filesystem.Copier             = (*fs)(nil)
)

View on GitHub (pinned to 12126d8942)