hashicorp/nomad · error
error writing to file
Error message
error writing to file %q: %w
What it means
streamAllocDir copies files from a previous allocation's data directory (snapshot tar stream read via a tar.Reader) into the new allocation's directory. This error wraps any failure from f.Write while extracting the tar entry, and includes the destination file name and the underlying OS error.
Solutions
- Check disk space and inode availability on the client's alloc_dir volume (df -h / df -i).
- Inspect the destination file path and its permissions/ownership; ensure the nomad user can write to alloc_dir.
- Verify no symlinks in the previous alloc's data escape the alloc directory (Nomad rejects these); clean up the old alloc dir.
- Retry the allocation migration after freeing space or fixing permissions; a failed migration will be rescheduled.
Example fix
// before (server-side fix is not possible; caller-side remedy) // client hits: error writing to file "/data/alloc/.../app.log": write ...: no space left on device // after $ df -h /data/alloc # free space or expand volume, then restart the failed allocation
Defensive patterns
Strategy: validation
Validate before calling
// before enabling alloc migration on a client, precheck the target volume
if err := checkWritable(conf.AllocDir); err != nil {
return fmt.Errorf("alloc dir not writable, migration would fail: %w", err)
}
func checkWritable(dir string) error {
probe := filepath.Join(dir, ".nomad-write-probe")
if err := os.WriteFile(probe, []byte("ok"), 0o600); err != nil { return err }
return os.Remove(probe)
} Prevention
- Monitor disk usage on alloc_dir volumes and alert before they fill
- Ensure consistent ownership/permissions of alloc_dir across node maintenance
- Audit old alloc dirs for symlinks that could escape and block migration
- Keep Nomad updated: symlink-escape protections and stream handling have been patched over releases
When it happens
Trigger: During alloc migration (migrateAllocDir), a tar entry read from the previous allocation's snapshot is written to the new alloc dir and the write fails — typically disk full, permission denied on the destination path, a bad symlink target escaping the alloc dir, or the destination file being closed/unwritable.
Common situations: Disk quota or out-of-space on the client's alloc_dir volume; corrupted or malicious symlink entries in a previous alloc's data (the BadSymlink tests exercise exactly this); read-only filesystem; file previously created with restrictive permissions by another user.
Understand the failure class
Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.
Related errors
- can't seek to offset
- cannot write nomad binary for install
- could not copy file
- could not open existing file
- could not stat file
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/906a10fd285a2603.
Report an issue: GitHub.
Appendix: source
Thrown at client/allocwatcher/alloc_watcher.go:677
return fmt.Errorf("error chmoding file %w", err)
}
// Can't change owner if not root or on Windows.
if euid == 0 {
if err := f.Chown(hdr.Uid, hdr.Gid); err != nil {
f.Close()
return fmt.Errorf("error chowning file %w", err)
}
}
// We write in chunks so that we can test if the client
// is still alive
for !canceled() {
n, err := tr.Read(buf)
if n > 0 && (err == nil || err == io.EOF) {
if _, err := f.Write(buf[:n]); err != nil {
f.Close()
return fmt.Errorf("error writing to file %q: %w", f.Name(), err)
}
}
if err != nil {
f.Close()
if err != io.EOF {
return fmt.Errorf("error reading snapshot: %w", err)
}
break
}
}
}
}
if canceled() {
return ctx.Err()
}View on GitHub (pinned to 482b49bf1a)