dgraph-io/dgraph · error

While renaming object in s3, copy failed

Error message

While renaming object in s3, copy failed

What it means

s3Handler.Rename implements rename as copy-then-delete. The copy step is retried up to 100 times at 1-second intervals; each individual CopyObject failure is wrapped as 'While renaming object in s3, copy failed' and surfaced after retries are exhausted. The object was NOT renamed; the source remains intact.

Source

Thrown at worker/backup_handler.go:411

	sw := &s3Writer{
		bucketName: h.bucketName,
		cerr:       make(chan error, 1),
	}
	sw.preader, sw.pwriter = io.Pipe()
	go sw.upload(h.mc, objectPath)
	return sw, nil
}

func (h *s3Handler) Rename(srcPath, dstPath string) error {
	srcPath = h.getObjectPath(srcPath)
	dstPath = h.getObjectPath(dstPath)
	src := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}
	dst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}
	// We try copying 100 times, if it still fails, then the user should manually rename.
	err := x.RetryUntilSuccess(100, time.Second, func() error {
		if _, err := h.mc.CopyObject(context.Background(), dst, src); err != nil {
			return errors.Wrapf(err, "While renaming object in s3, copy failed")
		}
		return nil
	})
	if err != nil {
		return err
	}

	err = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})
	return errors.Wrap(err, "Rename failed to remove temporary file")
}

func (h *s3Handler) getObjectPath(path string) string {
	return filepath.Join(h.objectPrefix, cleanRelPath(path))
}

View on GitHub (pinned to 759e242be6)

Solutions

1. Verify the source object exists at srcPath (getObjectPath + prefix) before renaming.
2. Confirm src and dst are in the same bucket — the handler only copies within h.bucketName.
3. Check IAM/KMS: grant s3:GetObject on src and s3:PutObject on dst; verify SSE-KMS key access.
4. For objects over 5GB use minio's ComposeObject (multipart copy).
5. After fixing the cause, rename is safe to re-attempt — copy is idempotent and source is untouched.

Example fix

// before
src := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}
dst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}
// after: validate source first
if _, err := h.mc.StatObject(ctx, h.bucketName, srcPath, minio.StatObjectOptions{}); err != nil {
    return errors.Wrapf(err, "rename source %s missing", srcPath)
}
src := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}
dst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}

srcPath := h.getObjectPath(src)
if _, err := h.mc.StatObject(ctx, h.bucketName, srcPath, minio.StatObjectOptions{}); err != nil {
    return fmt.Errorf("rename source %s missing: %v", srcPath, err)
}

err := handler.Rename(tmp, final)
if err != nil {
    if strings.Contains(err.Error(), "While renaming object in s3, copy failed") {
        log.Printf("rename copy failed after retries; source intact at %s: %v", tmp, err)
        // source still exists: safe to retry or clean up manually
    }
    return err
}

When it happens

Trigger: minio CopyObject repeatedly fails for ~100 seconds: source object missing, insufficient permissions (s3:GetObject/s3:PutObject), SSE/KMS errors, object too large for a single copy, or persistent endpoint failure. Both CopySrc and CopyDest use h.bucketName — cross-bucket renames always fail.

Common situations: Cross-bucket rename attempt (both src and dst hardcoded to h.bucketName); source temp object not uploaded yet; KMS key unavailable; object >5GB copied without multipart ComposeObject; IAM policy missing GetObject on the source.

Related errors

• Failed to read s3 object
• Failed to read the s3 object
• Rename failed to remove temporary file
• Invalid bucket: %q
• Minio handler requires a host

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/ab2cc3c054a429a3. Report an issue: GitHub.