What causes "Rename failed to remove temporary file" in dgraph-io/dgraph?

Verify s3:DeleteObject permission on the bucket/prefix for the worker credentials.

Rename failed to remove temporary file

Error message

Rename failed to remove temporary file

What it means

After a successful S3 copy, s3Handler.Rename deletes the source object; if RemoveObject fails the error is wrapped as 'Rename failed to remove temporary file'. The copy already succeeded, so the data exists at the destination, but the source (typically a temporary object) is left behind — a cleanup failure, not data loss.

Source

Thrown at worker/backup_handler.go:420

func (h *s3Handler) Rename(srcPath, dstPath string) error {
	srcPath = h.getObjectPath(srcPath)
	dstPath = h.getObjectPath(dstPath)
	src := minio.CopySrcOptions{Bucket: h.bucketName, Object: srcPath}
	dst := minio.CopyDestOptions{Bucket: h.bucketName, Object: dstPath}
	// We try copying 100 times, if it still fails, then the user should manually rename.
	err := x.RetryUntilSuccess(100, time.Second, func() error {
		if _, err := h.mc.CopyObject(context.Background(), dst, src); err != nil {
			return errors.Wrapf(err, "While renaming object in s3, copy failed")
		}
		return nil
	})
	if err != nil {
		return err
	}

	err = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})
	return errors.Wrap(err, "Rename failed to remove temporary file")
}

func (h *s3Handler) getObjectPath(path string) string {
	return filepath.Join(h.objectPrefix, cleanRelPath(path))
}

View on GitHub (pinned to 759e242be6)

Solutions

Verify s3:DeleteObject permission on the bucket/prefix for the worker credentials.
Check for Object Lock/retention or bucket policies blocking deletion; use governance-mode bypass if authorized.
Re-attempt the rename or delete the source object manually (mc rm) — destination data is already safe.
Add a retry around RemoveObject and/or run periodic garbage collection of orphaned temp objects.
Check the wrapped minio error code to distinguish auth vs transient failures.

Example fix

// before
err = h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})
return errors.Wrap(err, "Rename failed to remove temporary file")
// after
err = x.RetryUntilSuccess(10, time.Second, func() error {
    return h.mc.RemoveObject(context.Background(), h.bucketName, srcPath, minio.RemoveObjectOptions{})
})
if err != nil {
    log.Printf("warning: rename done, stale source %s needs GC", srcPath)
}
return nil

Defensive patterns

Strategy: try-catch

Try / catch

err := handler.Rename(tmp, final)
if err != nil && strings.Contains(err.Error(), "Rename failed to remove temporary file") {
    log.Printf("WARN: destination written; stale temp object %s remains — schedule cleanup", tmp)
    return nil // data is safe at destination
}

Prevention

Grant DeleteObject alongside Put/Get for backup prefixes.
Avoid Object Lock on temp/staging prefixes.
Run periodic GC that lists the tmp prefix and deletes orphans older than N hours.
Wrap cleanup deletes in retries; treat them as best-effort, not fatal.

When it happens

Trigger: minio RemoveObject returns an error after the copy succeeded: s3:DeleteObject permission missing, object locked (Object Lock/retention), versioned bucket with deny-delete policy, or a transient endpoint error on this single (un-retried) delete call.

Common situations: IAM policy grants write but not delete; S3 Object Lock in compliance mode preventing deletion of the temp object; versioned buckets with delete protection; transient 5xx on the single RemoveObject call.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/7347dfcd68f291d3. Report an issue: GitHub.