apache/beam · error

StateChannel[%v].Send(%v): channel closed due to: %v

Error message

StateChannel[%v].Send(%v): channel closed due to: %v

What it means

StateChannel.Send fails when the channel's closedErr is set — the underlying gRPC stream to the state service has terminated (recv loop got an error/EOF and closed the channel). Send refuses new requests and reports the original close reason.

Source

Thrown at sdks/go/pkg/beam/core/runtime/harness/statemgr.go:786

	c.mu.Unlock()
	// Clean up everything else, this stream is done.
	c.terminateStreamOnError(err)

	if ok {
		ch <- &fnpb.StateResponse{Id: id, Error: fmt.Sprintf("StateChannel[%v].write failed to send: %v", c.id, err)}
	}
}

// Send sends a state request and returns the response.
func (c *StateChannel) Send(req *fnpb.StateRequest) (*fnpb.StateResponse, error) {
	id := fmt.Sprintf("r%v", atomic.AddInt32(&c.nextRequestNo, 1))
	req.Id = id

	ch := make(chan *fnpb.StateResponse, 1)
	c.mu.Lock()
	if c.closedErr != nil {
		defer c.mu.Unlock()
		return nil, errors.Wrapf(c.closedErr, "StateChannel[%v].Send(%v): channel closed due to: %v", c.id, id, c.closedErr)
	}
	c.responses[id] = ch
	c.mu.Unlock()

	c.requests <- req

	var resp *fnpb.StateResponse
	select {
	case resp = <-ch:
	case <-c.DoneCh:
		c.mu.Lock()
		defer c.mu.Unlock()
		return nil, errors.Wrapf(c.closedErr, "StateChannel[%v].Send(%v): context canceled", c.id, id)
	}
	if resp.Error != "" {
		return nil, errors.New(resp.Error)
	}
	return resp, nil

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped closedErr for the root cause (EOF, deadline, reset) and address that first.
2. Check network stability and any load-balancer/proxy idle timeouts between worker and runner; raise idle timeouts.
3. Reduce bundle duration or state request volume if the runner is dropping overloaded streams.
4. Retry the bundle/job; if the runner crashed, scale or fix the runner and rerun.

resp, err := w.Append(ctx, val)
if err != nil && strings.Contains(err.Error(), "channel closed due to") {
    return fmt.Errorf("state stream dropped mid-bundle: %w", err) // surface for runner-level retry
}

When it happens

Trigger: Any state read/write request issued after the channel's receive loop ended: runner closed the stream, network error, connection reset, or the server sent an error causing channel close.

Common situations: Long-running bundles where the state stream times out or is dropped, runner restarts mid-bundle, network instability between worker and runner, concurrent close during shutdown racing a send.

Related errors

• failed to send chunks for %v; close error: %v
• chunk send failed
• failed to connect to state service %v
• failed to connect: %v
• StageFile chunk send failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/062ce1a1d8bc0b6a. Report an issue: GitHub.