subscription closed by server, client should resubscribe

Error message

subscription closed by server, client should resubscribe

What it means

ErrSubscriptionClosed is returned by stream.Subscription.Next (and NextNoBlock) when the server has closed the subscription — for example because the ACL token's policies changed or the broker is shutting down. The caller must Unsubscribe and issue a fresh Subscribe request; the stream cannot be resumed.

Source

Thrown at nomad/stream/subscription.go:28

	"sync/atomic"

	"github.com/hashicorp/nomad/nomad/structs"
)

const (
	// subscriptionStateOpen is the default state of a subscription. An open
	// subscription may receive new events.
	subscriptionStateOpen uint32 = 0

	// subscriptionStateClosed indicates that the subscription was closed, possibly
	// as a result of a change to an ACL token, and will not receive new events.
	// The subscriber must issue a new Subscribe request.
	subscriptionStateClosed uint32 = 1
)

// ErrSubscriptionClosed is a error signalling the subscription has been
// closed. The client should Unsubscribe, then re-Subscribe.
var ErrSubscriptionClosed = errors.New("subscription closed by server, client should resubscribe")

type Subscription struct {
	// state must be accessed atomically 0 means open, 1 means closed with reload
	state atomic.Uint32

	req *SubscribeRequest

	// currentItem stores the current buffer item we are on. It
	// is mutated by calls to Next.
	currentItem *bufferItem

	// forceClosed is closed when forceClose is called. It is used by
	// EventBroker to cancel Next().
	forceClosed chan struct{}

	// unsub is a function set by EventBroker that is called to free resources
	// when the subscription is no longer needed.
	// It must be safe to call the function from multiple goroutines and the function

View on GitHub (pinned to 482b49bf1a)

Solutions

Detect ErrSubscriptionClosed from Next and immediately re-Subscribe to the broker
Unsubscribe the stale subscription before creating the new one
Use a reconnect loop with backoff around Next that resubscribes on this error
Keep the ACL token's policies stable, or expect and handle resubscription when policies change

Example fix

// before
for {
    ev, err := sub.Next(ctx)
    if err != nil { return err }
    handle(ev)
}
// after
for {
    ev, err := sub.Next(ctx)
    if errors.Is(err, stream.ErrSubscriptionClosed) {
        sub.Unsubscribe()
        sub, err = broker.Subscribe(&stream.SubscribeRequest{...})
        continue
    }
    if err != nil { return err }
    handle(ev)
}

Defensive patterns

Strategy: retry

Validate before calling

// No pre-call validation exists: the server closes the stream externally.
// Guard instead by checking state before Next where accessible:
if sub == nil { return errors.New("subscription is nil") }

Type guard

func subscriptionOpen(sub *stream.Subscription) bool {
    return sub != nil // closed state surfaces only via Next returning ErrSubscriptionClosed
}

Try / catch

ev, err := sub.Next(ctx)
if errors.Is(err, stream.ErrSubscriptionClosed) {
    sub.Unsubscribe()
    return resubscribe(ctx) // fresh Subscribe with backoff
}

Prevention

Wrap all Next loops in a resubscribe-with-backoff driver
Unsubscribe before re-Subscribing to avoid leaked subscriptions
Expect closure when the token's ACL policies change; keep a resubscribe path
Test consumers against broker shutdown to verify reconnect logic

When it happens

Trigger: Calling Next on a Subscription whose state flag is subscriptionStateClosed; the event broker closes subscriptions when the subscribing token's ACL policies are updated or when the broker shuts down.

Common situations: An event stream consumer's token had its policy list changed server-side mid-stream; the server is shutting down while a client still consumes events; a long-lived stream consumer does not resubscribe.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c0e5936bb3b254fc. Report an issue: GitHub.