argoproj/argo-workflows · error

plugin connection shutdown (socket= )

Error message

plugin %s connection shutdown (socket=%q)

What it means

After dialing, NewDriver waits for the gRPC channel to reach connectivity.Ready. If the channel enters connectivity.Shutdown — meaning the connection was closed (including the driver's own conn.Close() on a prior failure path) — construction aborts with this error. It signals the gRPC channel is dead and cannot become ready.

Solutions

1. Verify the plugin process is actually running and serving on the socket (ss -x | grep <socket> or attempt a grpc probe)
2. Remove stale socket files left by crashed plugin instances and restart the plugin
3. Check plugin logs for a panic/exit right after binding the socket
4. Ensure no code path calls Driver.Close() concurrently with NewDriver (common in tests with shared drivers)

Example fix

// before: stale socket from previous run, no listener
// after: clean up before plugin start
// rm -f /var/run/argo/plugins/myplug.sock && ./my-plugin --socket /var/run/argo/plugins/myplug.sock

// confirm something is listening before dialing
if conn, err := net.Dial("unix", socketPath); err != nil {
    return fmt.Errorf("nothing listening on %s: %w", socketPath, err)
} else { conn.Close() }

if strings.Contains(err.Error(), "connection shutdown") {
    // channel is closed: never reuse this Driver; rebuild it
    drv.Close()
    drv, err = plugin.NewDriver(ctx, name, path, timeout)
}

When it happens

Trigger: The channel transitions to Shutdown while waiting for Ready: the ClientConn was closed by another goroutine, grpc-go gave up reconnecting, or the transport was shut down due to repeated failures against the unix socket.

Common situations: Plugin process exits immediately after creating its socket (crash after bind), socket file exists but nothing is listening (stale socket left over from a previous run), concurrent Close() of the same Driver in tests.

Related errors

• failed to dial plugin
• timeout waiting for plugin
• plugin cannot stat unix socket at
• plugin context cancelled while waiting for socket at
• plugin expected unix socket at but it does not exist after…

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a12ae880e585d88d. Report an issue: GitHub.

Appendix: source

Thrown at workflow/artifacts/plugin/plugin.go:124

	}

	driver := &Driver{
		pluginName: pluginName,
		conn:       conn,
		client:     artifact.NewArtifactServiceClient(conn),
	}

	// Verify the connection by checking the connection state
	ctx, cancel := context.WithTimeout(ctx, connectionTimeout)
	defer cancel()

	conn.Connect()

	// Wait for the connection to reach Ready state within the timeout
	for state := conn.GetState(); state != connectivity.Ready; state = conn.GetState() {
		if state == connectivity.Shutdown {
			_ = conn.Close()
			return nil, fmt.Errorf("plugin %s connection shutdown (socket=%q)", pluginName, socketPath)
		}
		if !conn.WaitForStateChange(ctx, state) {
			// Timeout or context cancelled
			currentState := conn.GetState()
			_ = conn.Close()
			return nil, fmt.Errorf("timeout waiting for plugin %s connection to become ready, last state: %s (socket=%q)", pluginName, currentState, socketPath)
		}
	}

	logger.Info(ctx, fmt.Sprintf("plugin %s: connected successfully to %q", pluginName, socketPath))
	return driver, nil
}

// Close closes the gRPC connection
func (d *Driver) Close() error {
	if d.conn != nil {
		return d.conn.Close()
	}

View on GitHub (pinned to 35bff19146)