What causes "failed to read plugin directory %q: %v" in hashicorp/nomad?

Verify plugin_dir exists, is a directory, and is readable by the Nomad agent user (ls -ld ).

failed to read plugin directory %q: %v

Error message

failed to read plugin directory %q: %v

What it means

This error is returned by the plugin loader's scan() function in Nomad's helper/pluginutils/loader package when Readdirnames(-1) fails on the configured plugin directory (plugin_dir). The directory was successfully opened, but reading the list of entry names failed (e.g. due to I/O errors, permission changes, or the directory being removed mid-scan). It wraps the underlying OS error for context.

Source

Thrown at helper/pluginutils/loader/init.go:243

	if l.pluginDir == "" {
		return nil, nil
	}

	// Capture the list of binaries in the plugins folder
	f, err := os.Open(l.pluginDir)
	if err != nil {
		// There are no plugins to scan
		if os.IsNotExist(err) {
			l.logger.Warn("skipping external plugins since plugin_dir doesn't exist")
			return nil, nil
		}

		return nil, fmt.Errorf("failed to open plugin directory %q: %v", l.pluginDir, err)
	}
	files, err := f.Readdirnames(-1)
	f.Close()
	if err != nil {
		return nil, fmt.Errorf("failed to read plugin directory %q: %v", l.pluginDir, err)
	}

	var plugins []os.FileInfo
	for _, f := range files {
		f = filepath.Join(l.pluginDir, f)
		s, err := os.Stat(f)
		if err != nil {
			return nil, fmt.Errorf("failed to stat file %q: %v", f, err)
		}
		if s.IsDir() {
			l.logger.Warn("skipping subdir in plugin folder", "subdir", f)
			continue
		}

		if !executable(f, s) {
			l.logger.Warn("skipping un-executable file in plugin folder", "file", f)
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

Verify plugin_dir exists, is a directory, and is readable by the Nomad agent user (ls -ld <plugin_dir>).
Check the wrapped %v OS error for the root cause (e.g. 'permission denied', 'input/output error', 'too many open files').
If the directory is on a network filesystem, remount it and ensure stable connectivity, or move plugins to local disk.
If file-descriptor exhaustion, raise ulimit -n for the Nomad agent.
Restart the Nomad agent after fixing the filesystem condition so scan() runs again.

Example fix

// before (agent config, unreadable dir)
plugin_dir = "/opt/nomad-plugins-root-only"
// after
# chown nomad:nomad /opt/nomad-plugins && chmod 755 /opt/nomad-plugins
plugin_dir = "/opt/nomad-plugins"

Defensive patterns

Strategy: validation

Validate before calling

// before starting the agent / before configuring plugin_dir
d, err := os.Open(pluginDir)
if err != nil { log.Fatalf("plugin_dir unreadable: %v", err) }
if _, err := d.Readdirnames(1); err != nil { log.Fatalf("plugin_dir not listable: %v", err) }
d.Close()

Try / catch

// Go: caller of Load() handles the returned error
plugins, err := loader.New(...).Load()
if err != nil {
	var perr *os.PathError
	if errors.As(err, &perr) {
		log.Printf("plugin dir I/O problem at %s: %v", perr.Path, perr.Err)
	}
}

Prevention

Provision plugin_dir with correct ownership/permissions in config management before the agent starts.
Keep plugin_dir on local, stable storage rather than flaky network mounts.
Raise the agent's file-descriptor ulimit.
Never mutate plugin_dir while the agent is (re)loading plugins.

When it happens

Trigger: os.File.Readdirnames(-1) returns an error after OpenDir succeeded on l.pluginDir; typically caused by filesystem I/O errors, the directory being deleted between open and read, permission revocation mid-scan, or resource limits (too many open files).

Common situations: plugin_dir mounted on a network volume (NFS/EFS) that dropped; plugin_dir removed or replaced by a symlink while Nomad runs; running under a security policy (SELinux/AppArmor) that blocks reads; ulimit -n exhaustion leaving a bad descriptor.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/78da8bc1b10fdcd0. Report an issue: GitHub.