timeout waiting for plugin %s connection to become ready, la

Error message

timeout waiting for plugin %s connection to become ready, last state: %s (socket=%q)

What it means

NewDriver gives the gRPC channel connectionTimeout to reach connectivity.Ready. WaitForStateChange returns false when the context deadline expires first; the driver then reports the last observed connectivity state. This is the standard 'plugin unreachable within timeout' failure.

Source

Thrown at workflow/artifacts/plugin/plugin.go:130

	}

	// Verify the connection by checking the connection state
	ctx, cancel := context.WithTimeout(ctx, connectionTimeout)
	defer cancel()

	conn.Connect()

	// Wait for the connection to reach Ready state within the timeout
	for state := conn.GetState(); state != connectivity.Ready; state = conn.GetState() {
		if state == connectivity.Shutdown {
			_ = conn.Close()
			return nil, fmt.Errorf("plugin %s connection shutdown (socket=%q)", pluginName, socketPath)
		}
		if !conn.WaitForStateChange(ctx, state) {
			// Timeout or context cancelled
			currentState := conn.GetState()
			_ = conn.Close()
			return nil, fmt.Errorf("timeout waiting for plugin %s connection to become ready, last state: %s (socket=%q)", pluginName, currentState, socketPath)
		}
	}

	logger.Info(ctx, fmt.Sprintf("plugin %s: connected successfully to %q", pluginName, socketPath))
	return driver, nil
}

// Close closes the gRPC connection
func (d *Driver) Close() error {
	if d.conn != nil {
		return d.conn.Close()
	}
	return nil
}

// Load implements ArtifactDriver.Load by calling the plugin service
func (d *Driver) Load(ctx context.Context, inputArtifact *wfv1.Artifact, path string) error {
	grpcArtifact := convertToGRPC(inputArtifact)

View on GitHub (pinned to 35bff19146)

Solutions

Increase connectionTimeoutSeconds in the plugin configuration
Verify the plugin is accepting connections on the socket (grpcurl or plugin health check)
Check plugin logs/CPU for hangs or deadlocks; restart the plugin pod
Confirm plugin and executor containers share the volume mount and network namespace where the socket lives
Capture the reported last state in the message: TransientFailure vs Connecting points to refused connections vs slow startup respectively

Example fix

// before
connectionTimeoutSeconds: 5
// after
connectionTimeoutSeconds: 60

Defensive patterns

Strategy: retry

Validate before calling

// ensure a live listener exists and timeout is sane
if conn, err := net.DialTimeout("unix", socketPath, 2*time.Second); err != nil {
    return fmt.Errorf("plugin not accepting connections on %s", socketPath)
} else { conn.Close() }
if timeout < 10*time.Second { timeout = 30 * time.Second }

Try / catch

if strings.Contains(err.Error(), "timeout waiting for plugin") {
    // parse last state; retry once on Connecting/TransientFailure
    return retryAfterDelay(5*time.Second, func() error {
        var e error
        drv, e = plugin.NewDriver(ctx, name, path, timeout)
        return e
    })
}

Prevention

Set connectionTimeoutSeconds >= 30 for plugins with slow startup
Keep plugin health checks and restart policies aggressive so hung processes are recycled
Share the socket volume correctly between plugin and executor containers

When it happens

Trigger: The socket file exists but the plugin is not accepting connections, the plugin is overloaded/hung, or connectionTimeoutSeconds is too small for the plugin to accept within it.

Common situations: Plugin process hung (deadlock) after binding the socket, socket backlog full, connectionTimeoutSeconds configured to a few seconds while the plugin takes longer to serve, plugin bound on a different network namespace (container isolation) so the executor can't reach the socket.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ee655815a65cee26. Report an issue: GitHub.