{"record":{"id":"7347dfcd68f291d3","repo":"dgraph-io/dgraph","slug":"rename-failed-to-remove-temporary-file","errorCode":null,"errorMessage":"Rename failed to remove temporary file","messagePattern":"Rename failed to remove temporary file","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"worker/backup_handler.go","lineNumber":420,"sourceCode":"\nfunc (h *s3Handler) Rename(srcPath, dstPath string) error {\n\tsrcPath = h.getObjectPath(srcPath)\n\tdstPath = h.getObjectPath(dstPath)\n\tsrc := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}\n\tdst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}\n\t// We try copying 100 times, if it still fails, then the user should manually rename.\n\terr := x.RetryUntilSuccess(100, time.Second, func() error {\n\t\tif _, err := h.mc.CopyObject(context.Background(), dst, src); err != nil {\n\t\t\treturn errors.Wrapf(err, \"While renaming object in s3, copy failed\")\n\t\t}\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})\n\treturn errors.Wrap(err, \"Rename failed to remove temporary file\")\n}\n\nfunc (h *s3Handler) getObjectPath(path string) string {\n\treturn filepath.Join(h.objectPrefix, cleanRelPath(path))\n}\n","sourceCodeStart":402,"sourceCodeEnd":426,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/worker/backup_handler.go#L402-L426","documentation":"After a successful S3 copy, s3Handler.Rename deletes the source object; if RemoveObject fails the error is wrapped as 'Rename failed to remove temporary file'. The copy already succeeded, so the data exists at the destination, but the source (typically a temporary object) is left behind — a cleanup failure, not data loss.","triggerScenarios":"minio RemoveObject returns an error after the copy succeeded: s3:DeleteObject permission missing, object locked (Object Lock/retention), versioned bucket with deny-delete policy, or a transient endpoint error on this single (un-retried) delete call.","commonSituations":"IAM policy grants write but not delete; S3 Object Lock in compliance mode preventing deletion of the temp object; versioned buckets with delete protection; transient 5xx on the single RemoveObject call.","solutions":["Verify s3:DeleteObject permission on the bucket/prefix for the worker credentials.","Check for Object Lock/retention or bucket policies blocking deletion; use governance-mode bypass if authorized.","Re-attempt the rename or delete the source object manually (mc rm) — destination data is already safe.","Add a retry around RemoveObject and/or run periodic garbage collection of orphaned temp objects.","Check the wrapped minio error code to distinguish auth vs transient failures."],"exampleFix":"// before\nerr = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})\nreturn errors.Wrap(err, \"Rename failed to remove temporary file\")\n// after\nerr = x.RetryUntilSuccess(10, time.Second, func() error {\n    return h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})\n})\nif err != nil {\n    log.Printf(\"warning: rename done, stale source %s needs GC\", srcPath)\n}\nreturn nil","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := handler.Rename(tmp, final)\nif err != nil && strings.Contains(err.Error(), \"Rename failed to remove temporary file\") {\n    log.Printf(\"WARN: destination written; stale temp object %s remains — schedule cleanup\", tmp)\n    return nil // data is safe at destination\n}","preventionTips":["Grant DeleteObject alongside Put/Get for backup prefixes.","Avoid Object Lock on temp/staging prefixes.","Run periodic GC that lists the tmp prefix and deletes orphans older than N hours.","Wrap cleanup deletes in retries; treat them as best-effort, not fatal."],"tags":["s3","minio","cleanup","permissions"],"backgroundTag":"s3-delete-object-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}