{"record":{"id":"ab2cc3c054a429a3","repo":"dgraph-io/dgraph","slug":"while-renaming-object-in-s3-copy-failed","errorCode":null,"errorMessage":"While renaming object in s3, copy failed","messagePattern":"While renaming object in s3, copy failed","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"worker/backup_handler.go","lineNumber":411,"sourceCode":"\n\tsw := &s3Writer{\n\t\tbucketName: h.bucketName,\n\t\tcerr:       make(chan error, 1),\n\t}\n\tsw.preader, sw.pwriter = io.Pipe()\n\tgo sw.upload(h.mc, objectPath)\n\treturn sw, nil\n}\n\nfunc (h *s3Handler) Rename(srcPath, dstPath string) error {\n\tsrcPath = h.getObjectPath(srcPath)\n\tdstPath = h.getObjectPath(dstPath)\n\tsrc := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}\n\tdst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}\n\t// We try copying 100 times, if it still fails, then the user should manually rename.\n\terr := x.RetryUntilSuccess(100, time.Second, func() error {\n\t\tif _, err := h.mc.CopyObject(context.Background(), dst, src); err != nil {\n\t\t\treturn errors.Wrapf(err, \"While renaming object in s3, copy failed\")\n\t\t}\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})\n\treturn errors.Wrap(err, \"Rename failed to remove temporary file\")\n}\n\nfunc (h *s3Handler) getObjectPath(path string) string {\n\treturn filepath.Join(h.objectPrefix, cleanRelPath(path))\n}\n","sourceCodeStart":393,"sourceCodeEnd":426,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/worker/backup_handler.go#L393-L426","documentation":"s3Handler.Rename implements rename as copy-then-delete. The copy step is retried up to 100 times at 1-second intervals; each individual CopyObject failure is wrapped as 'While renaming object in s3, copy failed' and surfaced after retries are exhausted. The object was NOT renamed; the source remains intact.","triggerScenarios":"minio CopyObject repeatedly fails for ~100 seconds: source object missing, insufficient permissions (s3:GetObject/s3:PutObject), SSE/KMS errors, object too large for a single copy, or persistent endpoint failure. Both CopySrc and CopyDest use h.bucketName — cross-bucket renames always fail.","commonSituations":"Cross-bucket rename attempt (both src and dst hardcoded to h.bucketName); source temp object not uploaded yet; KMS key unavailable; object >5GB copied without multipart ComposeObject; IAM policy missing GetObject on the source.","solutions":["Verify the source object exists at srcPath (getObjectPath + prefix) before renaming.","Confirm src and dst are in the same bucket — the handler only copies within h.bucketName.","Check IAM/KMS: grant s3:GetObject on src and s3:PutObject on dst; verify SSE-KMS key access.","For objects over 5GB use minio's ComposeObject (multipart copy).","After fixing the cause, rename is safe to re-attempt — copy is idempotent and source is untouched."],"exampleFix":"// before\nsrc := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}\ndst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}\n// after: validate source first\nif _, err := h.mc.StatObject(ctx, h.bucketName, srcPath, minio.StatObjectOptions{}); err != nil {\n    return errors.Wrapf(err, \"rename source %s missing\", srcPath)\n}\nsrc := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}\ndst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}","handlingStrategy":"retry","validationCode":"srcPath := h.getObjectPath(src)\nif _, err := h.mc.StatObject(ctx, h.bucketName, srcPath, minio.StatObjectOptions{}); err != nil {\n    return fmt.Errorf(\"rename source %s missing: %v\", srcPath, err)\n}","typeGuard":null,"tryCatchPattern":"err := handler.Rename(tmp, final)\nif err != nil {\n    if strings.Contains(err.Error(), \"While renaming object in s3, copy failed\") {\n        log.Printf(\"rename copy failed after retries; source intact at %s: %v\", tmp, err)\n        // source still exists: safe to retry or clean up manually\n    }\n    return err\n}","preventionTips":["Rename only within the same bucket/prefix this handler supports.","Stat the source before Rename; write temp objects atomically then rename.","Grant copy permissions for the whole prefix to the worker's credentials.","Use ComposeObject for very large objects instead of single CopyObject."],"tags":["s3","minio","rename","retry"],"backgroundTag":"s3-copy-object-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}