What causes "Couldn't copy %q to %q: %w" in hashicorp/nomad?

Inspect the wrapped error (%w) for the root cause and free disk space if it indicates ENOSPC.

Couldn't copy %q to %q: %w

Error message

Couldn't copy %q to %q: %w

What it means

This error is returned by fileCopy in the allocdir package when io.Copy fails to copy the source file's contents into the newly created destination file. HashiCorp Nomad wraps the underlying OS error with both the source and destination paths so the operator can tell which file transfer inside an allocation directory failed. It is a generic wrapper: the real cause (disk full, I/O error, permission denied, etc.) is in the wrapped error.

Source

Thrown at client/allocdir/alloc_dir.go:579

// fileCopy from src to dst setting the permissions and owner (if uid & gid are
// both greater than 0)
func fileCopy(src, dst string, uid, gid int, perm os.FileMode) error {
	// Do a simple copy.
	srcFile, err := os.Open(src)
	if err != nil {
		return fmt.Errorf("Couldn't open src file %v: %w", src, err)
	}
	defer srcFile.Close()

	dstFile, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE, perm)
	if err != nil {
		return fmt.Errorf("Couldn't create destination file %v: %w", dst, err)
	}
	defer dstFile.Close()

	if _, err := io.Copy(dstFile, srcFile); err != nil {
		return fmt.Errorf("Couldn't copy %q to %q: %w", src, dst, err)
	}

	if uid != idUnsupported && gid != idUnsupported {
		if err := dstFile.Chown(uid, gid); err != nil {
			return fmt.Errorf("Couldn't copy %q to %q: %w", src, dst, err)
		}
	}

	return nil
}

// pathExists is a helper function to check if the path exists.
func pathExists(path string) bool {
	if _, err := os.Stat(path); err != nil {
		if os.IsNotExist(err) {
			return false
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped error (%w) for the root cause and free disk space if it indicates ENOSPC.
Check read permissions on the source file and write permissions on the destination directory.
Verify the health of the underlying filesystem/volume (dmesg, SMART) if I/O errors are reported.
Retry the allocation; linkOrCopy falls back to copying when hard links fail, so transient FS issues may resolve.

Example fix

// before: opaque failure with only paths
return fmt.Errorf("Couldn't copy %q to %q: %w", src, dst, err)
// after (caller side): surface wrapped cause and retry transient failures
if err := linkOrCopy(src, dst, perm, uid, gid); err != nil {
    if errors.Is(err, syscall.ENOSPC) {
        return fmt.Errorf("alloc dir full; free space and retry: %w", err)
    }
    return err
}

Defensive patterns

Strategy: try-catch

Validate before calling

// before calling the copy path
if st, err := os.Stat(src); err != nil || st.IsDir() {
    return fmt.Errorf("source unreadable or not a file: %s", src)
}
if free, err := diskFree(filepath.Dir(dst)); err == nil && free < st.Size() {
    return fmt.Errorf("insufficient space for %s", dst)
}

Try / catch

if err := linkOrCopy(src, dst, perm, uid, gid); err != nil {
    var perr *fs.PathError
    if errors.As(err, &perr) {
        log.Printf("copy failed %s->%s: op=%s path=%s: %v", src, dst, perr.Op, perr.Path, perr.Err)
    }
    if errors.Is(err, syscall.ENOSPC) { /* free space / retry later */ }
    return err
}

Prevention

Monitor free space on the client data_dir volume and alert before it fills.
Keep alloc dir on a single healthy local filesystem.
Ensure the client user can read all sources it is asked to copy.

When it happens

Trigger: fileCopy(src, dst, perm, uid, gid) was invoked via linkOrCopy after a hard link was not possible, the destination file was created successfully, but the subsequent io.Copy(dstFile, srcFile) returned an error (read error on src or write error on dst).

Common situations: Disk full on the host or volume holding the allocation directory; source file unreadable due to permission changes; I/O errors on failing disks; NFS/remote filesystem failures when the alloc dir is on shared storage; copying large files that exceed quota.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6a3dec060fc78c99. Report an issue: GitHub.