argoproj/argo-workflows · error

plugin %s cannot stat unix socket at %q: %w

Error message

plugin %s cannot stat unix socket at %q: %w

What it means

NewDriver verifies the plugin's unix socket file exists before dialing. If os.Stat fails with an error that is NOT 'file not found' (e.g. permission denied on a parent directory, or a path component is not a directory), the driver fails immediately instead of retrying, wrapping the underlying stat error.

Source

Thrown at workflow/artifacts/plugin/plugin.go:52

	// Check for the unix socket, retrying for up to two minutes if it doesn't exist immediately
	logger := logging.RequireLoggerFromContext(ctx)

	// Try for up to 120 seconds, checking once per second
	const maxRetries = 120
	var info os.FileInfo
	var statErr error
	var socketExists bool

	for retry := range maxRetries {
		info, statErr = os.Stat(socketPath)
		if statErr == nil {
			socketExists = true
			break
		}

		if !os.IsNotExist(statErr) {
			// If error is not due to missing file, fail immediately
			return nil, fmt.Errorf("plugin %s cannot stat unix socket at %q: %w", pluginName, socketPath, statErr)
		}

		// Socket doesn't exist yet, log at debug level and retry
		logger.WithFields(logging.Fields{
			"pluginName": pluginName,
			"socketPath": socketPath,
			"retry":      retry,
			"maxRetries": maxRetries,
		}).Debug(ctx, "plugin socket not found, retrying in 1s")

		// Use context-aware sleep
		select {
		case <-time.After(time.Second):
			// Continue to next iteration
		case <-ctx.Done():
			return nil, fmt.Errorf("plugin %s context cancelled while waiting for socket at %q: %w", pluginName, socketPath, ctx.Err())
		}
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check permissions on every directory component of socketPath; make them readable/searchable by the workflow executor user
2. Verify the socketPath is a directory path plus socket filename, not a path that runs through a regular file
3. Confirm the plugin pod's emptyDir/hostPath volume is mounted at the expected location (kubectl exec and ls -la the directory)
4. If the filesystem is flaky, increase logging and inspect the wrapped statErr (e.g. 'permission denied' vs 'not a directory') to target the fix

Example fix

// before: socketPath = "/var/run/argo/plugins/myplug.sock" but /var/run/argo/plugins is 0700 root-only
// after: ensure directory perms
// chmod 755 /var/run/argo/plugins
// or in the plugin container spec:
// securityContext: { runAsUser: 8737 }

if info, err := os.Stat(socketPath); err != nil {
    if !os.IsNotExist(err) {
        return fmt.Errorf("socket path %q unusable before creating plugin driver: %w", socketPath, err)
    }
} else if dir := filepath.Dir(socketPath); dir != "." {
    if _, err := os.Stat(dir); err != nil || !isSearchable(dir) {
        return fmt.Errorf("socket directory %q not accessible", dir)
    }
}

When it happens

Trigger: Calling NewDriver with a socketPath whose parent directory is unreadable/executable-by-permission, the path traverses a nonexistent directory component (ENOTDIR), the path is too long (ENAMETOOLONG), or filesystem/I/O errors while statting — anything other than ENOENT.

Common situations: Wrong socketPath typo pointing at a regular file inside an inaccessible directory, container volume mounts where the plugin's socket directory has wrong ownership/permissions, or the path contains a file where a directory is expected after a config change.

Related errors

• failed to create folder path: %w
• failed to write large arg %d to file: %w
• failed to leave working directory before staging input artif
• failed to create parent directory for artifact %q at %s: %w
• failed to clear existing path for artifact %q at %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/d145ec40eed0efe2. Report an issue: GitHub.