apache/beam · error

error parsing S3 source uri

Error message

error parsing S3 source uri %s: %v

What it means

Returned by fs.Copy when the SOURCE path (oldpath) cannot be parsed as a valid S3 URI. parseURI requires an s3://bucket/key form; invalid schemes or missing components are rejected before the copy operation.

Solutions

1. Check that oldpath is a well-formed s3://bucket/key URI.
2. Fix path-building code that drops or duplicates the scheme or slashes.
3. Use the appropriate filesystem for non-S3 sources instead of fs.Copy.

Example fix

// before
fsys.Copy(ctx, "bucket/key", "s3://bucket/key2") // missing scheme

// after
src, dst := "s3://bucket/key", "s3://bucket/key2"
if _, _, err := parseURIPublic(src); err != nil { return fmt.Errorf("bad source %q: %w", src, err) }
fsys.Copy(ctx, src, dst)

func validS3URI(p string) error {
    if !strings.HasPrefix(p, "s3://") { return fmt.Errorf("missing s3:// scheme: %q", p) }
    rest := strings.TrimPrefix(p, "s3://")
    bucket, key, ok := strings.Cut(rest, "/")
    if !ok || bucket == "" || key == "" { return fmt.Errorf("need s3://bucket/key, got %q", p) }
    return nil
}

if err := validS3URI(oldpath); err != nil { return fmt.Errorf("copy source invalid: %w", err) }
if err := fsys.Copy(ctx, oldpath, newpath); err != nil { return err }

When it happens

Trigger: Calling fs.Copy(ctx, oldpath, newpath) where oldpath is not a valid S3 URI (e.g. "s3:/bucket/key", local path, or "s3://" with empty bucket/key).

Common situations: Copy jobs mixing local and s3 paths; string formatting bugs producing malformed URIs; copying from GCS into S3 through the S3 filesystem.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• error parsing S3 destination uri
• error parsing S3 uri
• error parsing S3 uri
• Param validation failed on part number
• All parts but the last must be larger than

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1a8cdf1621d4f67b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:212

		return fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.DeleteObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	if _, err = f.client.DeleteObject(ctx, params); err != nil {
		return fmt.Errorf("error deleting object %s: %v", filename, err)
	}

	return nil
}

// Copy copies the file from the old path to the new path.
func (f *fs) Copy(ctx context.Context, oldpath, newpath string) error {
	sourceBucket, sourceKey, err := parseURI(oldpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 source uri %s: %v", oldpath, err)
	}

	copySource := fmt.Sprintf("%s/%s", sourceBucket, sourceKey)
	destBucket, destKey, err := parseURI(newpath)
	if err != nil {
		return fmt.Errorf("error parsing S3 destination uri %s: %v", newpath, err)
	}

	params := &s3.CopyObjectInput{
		Bucket:     aws.String(destBucket),
		CopySource: aws.String(copySource),
		Key:        aws.String(destKey),
	}
	if _, err = f.client.CopyObject(ctx, params); err != nil {
		return fmt.Errorf("error copying object %s: %v", oldpath, err)
	}

	return nil

View on GitHub (pinned to 12126d8942)