argoproj/argo-workflows · error

plugin %s connection shutdown (socket=%q)

Error message

plugin %s connection shutdown (socket=%q)

What it means

After dialing, NewDriver waits for the gRPC channel to reach connectivity.Ready. If the channel enters connectivity.Shutdown — meaning the connection was closed (including the driver's own conn.Close() on a prior failure path) — construction aborts with this error. It signals the gRPC channel is dead and cannot become ready.

Source

Thrown at workflow/artifacts/plugin/plugin.go:124

	}

	driver := &Driver{
		pluginName: pluginName,
		conn:       conn,
		client:     artifact.NewArtifactServiceClient(conn),
	}

	// Verify the connection by checking the connection state
	ctx, cancel := context.WithTimeout(ctx, connectionTimeout)
	defer cancel()

	conn.Connect()

	// Wait for the connection to reach Ready state within the timeout
	for state := conn.GetState(); state != connectivity.Ready; state = conn.GetState() {
		if state == connectivity.Shutdown {
			_ = conn.Close()
			return nil, fmt.Errorf("plugin %s connection shutdown (socket=%q)", pluginName, socketPath)
		}
		if !conn.WaitForStateChange(ctx, state) {
			// Timeout or context cancelled
			currentState := conn.GetState()
			_ = conn.Close()
			return nil, fmt.Errorf("timeout waiting for plugin %s connection to become ready, last state: %s (socket=%q)", pluginName, currentState, socketPath)
		}
	}

	logger.Info(ctx, fmt.Sprintf("plugin %s: connected successfully to %q", pluginName, socketPath))
	return driver, nil
}

// Close closes the gRPC connection
func (d *Driver) Close() error {
	if d.conn != nil {
		return d.conn.Close()
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify the plugin process is actually running and serving on the socket (ss -x | grep <socket> or attempt a grpc probe)
2. Remove stale socket files left by crashed plugin instances and restart the plugin
3. Check plugin logs for a panic/exit right after binding the socket
4. Ensure no code path calls Driver.Close() concurrently with NewDriver (common in tests with shared drivers)

Example fix

// before: stale socket from previous run, no listener
// after: clean up before plugin start
// rm -f /var/run/argo/plugins/myplug.sock && ./my-plugin --socket /var/run/argo/plugins/myplug.sock

// confirm something is listening before dialing
if conn, err := net.Dial("unix", socketPath); err != nil {
    return fmt.Errorf("nothing listening on %s: %w", socketPath, err)
} else { conn.Close() }

if strings.Contains(err.Error(), "connection shutdown") {
    // channel is closed: never reuse this Driver; rebuild it
    drv.Close()
    drv, err = plugin.NewDriver(ctx, name, path, timeout)
}

When it happens

Trigger: The channel transitions to Shutdown while waiting for Ready: the ClientConn was closed by another goroutine, grpc-go gave up reconnecting, or the transport was shut down due to repeated failures against the unix socket.

Common situations: Plugin process exits immediately after creating its socket (crash after bind), socket file exists but nothing is listening (stale socket left over from a previous run), concurrent Close() of the same Driver in tests.

Related errors

• failed to dial plugin %s at %q: %w
• timeout waiting for plugin %s connection to become ready, la
• plugin %s cannot stat unix socket at %q: %w
• plugin %s context cancelled while waiting for socket at %q:
• plugin %s expected unix socket at %q but it does not exist a

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a12ae880e585d88d. Report an issue: GitHub.