hashicorp/nomad · error

failed to create staging directory for volume (%s): %v

Error message

failed to create staging directory for volume (%s): %v

What it means

ensureStagingDir in Nomad's CSI volume manager creates the host staging directory (mode 0700) for a volume before StageVolume. If os.MkdirAll fails with an error other than 'already exists', it fails staging with this message wrapping the OS error.

Source

Thrown at client/pluginmanager/csimanager/volume.go:441

	return filepath.Join(root, AllocSpecificDirName, allocID, volID)
}

func (v *volumeManager) targetForVolume(root string, volID, allocID string, usage *UsageOptions) string {
	return filepath.Join(root, AllocSpecificDirName, allocID, volID, usage.ToFS())
}

// ensureStagingDir attempts to create a directory for use when staging a volume
// and then validates that the path is not already a mount point for e.g an
// existing volume stage.
//
// Returns whether the directory is a pre-existing mountpoint, the staging path,
// and any errors that occurred.
func (v *volumeManager) ensureStagingDir(vol *structs.CSIVolume, usage *UsageOptions) (string, bool, error) {
	hostStagingPath := v.stagingDirForVolume(v.mountRoot, vol.Namespace, vol.ID, usage)

	// Make the staging path, owned by the Nomad User
	if err := os.MkdirAll(hostStagingPath, 0700); err != nil && !os.IsExist(err) {
		return "", false, fmt.Errorf("failed to create staging directory for volume (%s): %v", vol.ID, err)

	}

	// Validate that it is not already a mount point
	m := mount.New()
	isNotMount, err := m.IsNotAMountPoint(hostStagingPath)
	if err != nil {
		return "", false, fmt.Errorf("mount point detection failed for volume (%s): %v", vol.ID, err)
	}

	return hostStagingPath, !isNotMount, nil
}

// ensureAllocDir attempts to create a directory for use when publishing a volume
// and then validates that the path is not already a mount point (e.g when reattaching
// to existing allocs).
//
// Returns whether the directory is a pre-existing mountpoint, the publish path,

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Check the host filesystem at the staging path: ensure parent directories exist, are writable by the Nomad user, and are directories not files
  2. Fix permissions (chown/chmod) or relocate the staging dir to a valid path and restart the Nomad client
  3. Free disk space or remount the filesystem read-write if the OS error indicates ENOSPC/EROFS
  4. Inspect the wrapped %v OS error to identify the exact cause before retrying stageVolume

Example fix

// before (host)
ls -l /var/lib/nomad/client/csi
staging -> /some/regular/file
// after (host)
rm /var/lib/nomad/client/csi/staging
mkdir -p /var/lib/nomad/client/csi/staging && chown nomad:nomad /var/lib/nomad/client/csi/staging
Defensive patterns

Strategy: validation

Validate before calling

stagingPath := "/var/lib/nomad/client/csi/staging"
fi, err := os.Stat(stagingPath)
if err == nil && !fi.IsDir() { return fmt.Errorf("%s is not a directory", stagingPath) }
if err := os.MkdirAll(stagingPath, 0700); err != nil { return err }

Try / catch

path, staged, err := vm.EnsureStagingDir(vol, usage)
if err != nil {
    return fmt.Errorf("staging dir setup failed: %w", err) // inspect wrapped OS error
}

Prevention

When it happens

Trigger: os.MkdirAll(hostStagingPath, 0700) returns a non-IsExist error — e.g. a parent path component is a file, permission denied on the mount root, disk full, or read-only filesystem — during stageVolume.

Common situations: Incorrect -clienthost-volume-staging-directory (mount root) permissions; staging dir path replaced by a regular file; SELinux/AppArmor or container security policy blocking mkdir under /var/lib/nomad; disk full on the node.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3178de6586c8ff7e. Report an issue: GitHub.