What causes "error removing existing file: %w" in hashicorp/nomad?

Inspect the destination path — if it is a directory, clear it or fix the archive layout

error removing existing file: %w

Error message

error removing existing file: %w

What it means

When extracting a regular file entry, Nomad removes any existing file at the destination path before creating a new one. This error is returned when os.Remove on the pre-existing path fails, e.g. due to permissions or the path being a non-empty directory.

Source

Thrown at client/allocwatcher/alloc_watcher.go:648

				return fmt.Errorf("error creating symlink: %w", err)
			}

			for _, path := range []string{hdr.Name, hdr.Linkname} {
				if escapes, err := escapingfs.PathEscapesAllocDir(dest, "", path); err != nil {
					return fmt.Errorf("error evaluating symlink: %w", err)
				} else if escapes {
					return fmt.Errorf("archive contains symlink that escapes alloc dir")
				}
			}

			continue
		}
		// If the header is a file, we write to a file
		if hdr.Typeflag == tar.TypeReg {
			fPath := filepath.Join(dest, hdr.Name)
			if _, err := os.Lstat(fPath); err == nil {
				if err := os.Remove(fPath); err != nil {
					return fmt.Errorf("error removing existing file: %w", err)
				}
			}
			f, err := os.Create(fPath)
			if err != nil {
				return fmt.Errorf("error creating file: %w", err)
			}

			// Setting the permissions of the file as the origin.
			if err := f.Chmod(os.FileMode(hdr.Mode)); err != nil {
				f.Close()
				return fmt.Errorf("error chmoding file %w", err)
			}

			// Can't change owner if not root or on Windows.
			if euid == 0 {
				if err := f.Chown(hdr.Uid, hdr.Gid); err != nil {
					f.Close()
					return fmt.Errorf("error chowning file %w", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the destination path — if it is a directory, clear it or fix the archive layout
Fix ownership/permissions on stale files in the alloc dir (run as root or chown manually)
Clean the partially extracted destination dir and retry the migration
Ensure the alloc dir is not mounted read-only

Defensive patterns

Strategy: validation

Validate before calling

// Before migration, check the destination dir has no stale path where a file is expected:
// if fi, err := os.Lstat(p); err == nil && fi.IsDir() { clean up or resolve conflict }

Try / catch

if err := watcher.Wait(ctx); err != nil {
    if strings.Contains(err.Error(), "error removing existing file") {
        os.RemoveAll(dest)
        return retryMigration(ctx, alloc)
    }
    return err
}

Prevention

Clean leftover state from failed extractions before retrying
Run the Nomad client with ownership of its data dir
Never mount the alloc dir read-only

When it happens

Trigger: os.Lstat(fPath) succeeds (something already exists) and os.Remove(fPath) returns an error during TypeReg extraction.

Common situations: Destination path is a non-empty directory where the archive expects a file (EISDIR/ENOTEMPTY); files owned by another UID; read-only mount; leftover state from a failed previous extraction.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/594781330301cbd4. Report an issue: GitHub.