hashicorp/nomad · error

failed to stat file %q: %v

Error message

failed to stat file %q: %v

What it means

In scan(), after listing directory entry names, each entry is joined with pluginDir and passed to os.Stat to get FileInfo. This error is returned when os.Stat fails for a specific file, wrapping the OS error. It means an entry listed in the plugin directory could not be stat'd, usually a race (file deleted between listing and stat) or a permission problem.

Source

Thrown at helper/pluginutils/loader/init.go:251

		if os.IsNotExist(err) {
			l.logger.Warn("skipping external plugins since plugin_dir doesn't exist")
			return nil, nil
		}

		return nil, fmt.Errorf("failed to open plugin directory %q: %v", l.pluginDir, err)
	}
	files, err := f.Readdirnames(-1)
	f.Close()
	if err != nil {
		return nil, fmt.Errorf("failed to read plugin directory %q: %v", l.pluginDir, err)
	}

	var plugins []os.FileInfo
	for _, f := range files {
		f = filepath.Join(l.pluginDir, f)
		s, err := os.Stat(f)
		if err != nil {
			return nil, fmt.Errorf("failed to stat file %q: %v", f, err)
		}
		if s.IsDir() {
			l.logger.Warn("skipping subdir in plugin folder", "subdir", f)
			continue
		}

		if !executable(f, s) {
			l.logger.Warn("skipping un-executable file in plugin folder", "file", f)
			continue
		}
		plugins = append(plugins, s)
	}

	return plugins, nil
}

// fingerprintPlugins fingerprints all external plugin binaries
func (l *PluginLoader) fingerprintPlugins(plugins []os.FileInfo, configs map[string]*config.PluginConfig) (map[PluginID]*pluginInfo, error) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v OS error to identify which file and why (e.g. 'no such file or directory' = dangling symlink/race).
2. Remove broken symlinks or stale entries from plugin_dir.
3. Ensure the Nomad agent user has read+execute (traverse) permission on plugin_dir and every path component.
4. Avoid mutating plugin_dir while the agent starts; perform plugin installs/upgrades with the agent stopped or atomically (write-then-rename).
5. Restart the agent and confirm plugins fingerprint.

Example fix

// before: broken symlink in plugin dir
/opt/nomad-plugins/docker -> /usr/local/bin/docker-plugin-removed
// after
rm /opt/nomad-plugins/docker  # then restart the agent

// audit plugin_dir for un-stat-able entries before agent start
entries, _ := os.ReadDir(pluginDir)
for _, e := range entries {
	p := filepath.Join(pluginDir, e.Name())
	if _, err := os.Stat(p); err != nil {
		log.Printf("removing unstat-able plugin entry %s: %v", p, err)
		os.Remove(p)
	}
}

// Go: surface the wrapped PathError per-file
if _, err := os.Stat(path); err != nil {
	var perr *os.PathError
	if errors.As(err, &perr) && errors.Is(perr.Err, fs.ErrNotExist) {
		// dangling symlink or raced delete: clean up and rescan
	}
}

When it happens

Trigger: os.Stat(f) returns an error for an entry in plugin_dir: dangling symlink (target removed after Readdirnames), file deleted concurrently, execute/read permission denied on parent path components, or overly long path names.

Common situations: Broken symlinks left in plugin_dir pointing to uninstalled plugin binaries; plugin upgrade script removing files while Nomad agent is starting; plugin dir containing entries the nomad user cannot traverse; immutable/locked files on Windows.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• failed to read plugin directory %q: %v
• plugin not found
• plugin not executable
• ErrPluginNotExists
• missing allocation ID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/331eccfc2658a521. Report an issue: GitHub.