{"record":{"id":"c0e5936bb3b254fc","repo":"hashicorp/nomad","slug":"subscription-closed-by-server-client-should-resub","errorCode":null,"errorMessage":"subscription closed by server, client should resubscribe","messagePattern":"subscription closed by server, client should resubscribe","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/stream/subscription.go","lineNumber":28,"sourceCode":"\t\"sync/atomic\"\n\n\t\"github.com/hashicorp/nomad/nomad/structs\"\n)\n\nconst (\n\t// subscriptionStateOpen is the default state of a subscription. An open\n\t// subscription may receive new events.\n\tsubscriptionStateOpen uint32 = 0\n\n\t// subscriptionStateClosed indicates that the subscription was closed, possibly\n\t// as a result of a change to an ACL token, and will not receive new events.\n\t// The subscriber must issue a new Subscribe request.\n\tsubscriptionStateClosed uint32 = 1\n)\n\n// ErrSubscriptionClosed is a error signalling the subscription has been\n// closed. The client should Unsubscribe, then re-Subscribe.\nvar ErrSubscriptionClosed = errors.New(\"subscription closed by server, client should resubscribe\")\n\ntype Subscription struct {\n\t// state must be accessed atomically 0 means open, 1 means closed with reload\n\tstate atomic.Uint32\n\n\treq *SubscribeRequest\n\n\t// currentItem stores the current buffer item we are on. It\n\t// is mutated by calls to Next.\n\tcurrentItem *bufferItem\n\n\t// forceClosed is closed when forceClose is called. It is used by\n\t// EventBroker to cancel Next().\n\tforceClosed chan struct{}\n\n\t// unsub is a function set by EventBroker that is called to free resources\n\t// when the subscription is no longer needed.\n\t// It must be safe to call the function from multiple goroutines and the function","sourceCodeStart":10,"sourceCodeEnd":46,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/stream/subscription.go#L10-L46","documentation":"ErrSubscriptionClosed is returned by stream.Subscription.Next (and NextNoBlock) when the server has closed the subscription — for example because the ACL token's policies changed or the broker is shutting down. The caller must Unsubscribe and issue a fresh Subscribe request; the stream cannot be resumed.","triggerScenarios":"Calling Next on a Subscription whose state flag is subscriptionStateClosed; the event broker closes subscriptions when the subscribing token's ACL policies are updated or when the broker shuts down.","commonSituations":"An event stream consumer's token had its policy list changed server-side mid-stream; the server is shutting down while a client still consumes events; a long-lived stream consumer does not resubscribe.","solutions":["Detect ErrSubscriptionClosed from Next and immediately re-Subscribe to the broker","Unsubscribe the stale subscription before creating the new one","Use a reconnect loop with backoff around Next that resubscribes on this error","Keep the ACL token's policies stable, or expect and handle resubscription when policies change"],"exampleFix":"// before\nfor {\n    ev, err := sub.Next(ctx)\n    if err != nil { return err }\n    handle(ev)\n}\n// after\nfor {\n    ev, err := sub.Next(ctx)\n    if errors.Is(err, stream.ErrSubscriptionClosed) {\n        sub.Unsubscribe()\n        sub, err = broker.Subscribe(&stream.SubscribeRequest{...})\n        continue\n    }\n    if err != nil { return err }\n    handle(ev)\n}","handlingStrategy":"retry","validationCode":"// No pre-call validation exists: the server closes the stream externally.\n// Guard instead by checking state before Next where accessible:\nif sub == nil { return errors.New(\"subscription is nil\") }","typeGuard":"func subscriptionOpen(sub *stream.Subscription) bool {\n    return sub != nil // closed state surfaces only via Next returning ErrSubscriptionClosed\n}","tryCatchPattern":"ev, err := sub.Next(ctx)\nif errors.Is(err, stream.ErrSubscriptionClosed) {\n    sub.Unsubscribe()\n    return resubscribe(ctx) // fresh Subscribe with backoff\n}","preventionTips":["Wrap all Next loops in a resubscribe-with-backoff driver","Unsubscribe before re-Subscribing to avoid leaked subscriptions","Expect closure when the token's ACL policies change; keep a resubscribe path","Test consumers against broker shutdown to verify reconnect logic"],"tags":["nomad","streaming","event-broker","subscription"],"backgroundTag":"subscription-closed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}