hashicorp/nomad · error

failed to create staging directory for volume

Error message

failed to create staging directory for volume (%s): %v

What it means

ensureStagingDir in Nomad's CSI volume manager creates the host staging directory (mode 0700) for a volume before StageVolume. If os.MkdirAll fails with an error other than 'already exists', it fails staging with this message wrapping the OS error.

Solutions

  1. Check the host filesystem at the staging path: ensure parent directories exist, are writable by the Nomad user, and are directories not files
  2. Fix permissions (chown/chmod) or relocate the staging dir to a valid path and restart the Nomad client
  3. Free disk space or remount the filesystem read-write if the OS error indicates ENOSPC/EROFS
  4. Inspect the wrapped %v OS error to identify the exact cause before retrying stageVolume

Example fix

// before (host)
ls -l /var/lib/nomad/client/csi
staging -> /some/regular/file
// after (host)
rm /var/lib/nomad/client/csi/staging
mkdir -p /var/lib/nomad/client/csi/staging && chown nomad:nomad /var/lib/nomad/client/csi/staging
Defensive patterns

Strategy: validation

Validate before calling

stagingPath := "/var/lib/nomad/client/csi/staging"
fi, err := os.Stat(stagingPath)
if err == nil && !fi.IsDir() { return fmt.Errorf("%s is not a directory", stagingPath) }
if err := os.MkdirAll(stagingPath, 0700); err != nil { return err }

Try / catch

path, staged, err := vm.EnsureStagingDir(vol, usage)
if err != nil {
    return fmt.Errorf("staging dir setup failed: %w", err) // inspect wrapped OS error
}

Prevention

When it happens

Trigger: os.MkdirAll(hostStagingPath, 0700) returns a non-IsExist error — e.g. a parent path component is a file, permission denied on the mount root, disk full, or read-only filesystem — during stageVolume.

Common situations: Incorrect -clienthost-volume-staging-directory (mount root) permissions; staging dir path replaced by a regular file; SELinux/AppArmor or container security policy blocking mkdir under /var/lib/nomad; disk full on the node.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3178de6586c8ff7e. Report an issue: GitHub.

Appendix: source

Thrown at client/pluginmanager/csimanager/volume.go:441

	return filepath.Join(root, AllocSpecificDirName, allocID, volID)
}

func (v *volumeManager) targetForVolume(root string, volID, allocID string, usage *UsageOptions) string {
	return filepath.Join(root, AllocSpecificDirName, allocID, volID, usage.ToFS())
}

// ensureStagingDir attempts to create a directory for use when staging a volume
// and then validates that the path is not already a mount point for e.g an
// existing volume stage.
//
// Returns whether the directory is a pre-existing mountpoint, the staging path,
// and any errors that occurred.
func (v *volumeManager) ensureStagingDir(vol *structs.CSIVolume, usage *UsageOptions) (string, bool, error) {
	hostStagingPath := v.stagingDirForVolume(v.mountRoot, vol.Namespace, vol.ID, usage)

	// Make the staging path, owned by the Nomad User
	if err := os.MkdirAll(hostStagingPath, 0700); err != nil && !os.IsExist(err) {
		return "", false, fmt.Errorf("failed to create staging directory for volume (%s): %v", vol.ID, err)

	}

	// Validate that it is not already a mount point
	m := mount.New()
	isNotMount, err := m.IsNotAMountPoint(hostStagingPath)
	if err != nil {
		return "", false, fmt.Errorf("mount point detection failed for volume (%s): %v", vol.ID, err)
	}

	return hostStagingPath, !isNotMount, nil
}

// ensureAllocDir attempts to create a directory for use when publishing a volume
// and then validates that the path is not already a mount point (e.g when reattaching
// to existing allocs).
//
// Returns whether the directory is a pre-existing mountpoint, the publish path,

View on GitHub (pinned to 482b49bf1a)