{"record":{"id":"93c45334fe267855","repo":"temporalio/temporal","slug":"controllerimpl-for-host-v-shutting-down","errorCode":null,"errorMessage":"ControllerImpl for host '%v' shutting down","messagePattern":"ControllerImpl for host '(.+?)' shutting down","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"service/history/shard/controller_impl.go","lineNumber":256,"sourceCode":"\n\t// Check again with exclusive lock\n\tif shard, ok := c.historyShards[shardID]; ok {\n\t\tif shard.IsValid() {\n\t\t\treturn shard, nil\n\t\t}\n\n\t\t// If the shard was invalid and still in the historyShards map, the\n\t\t// shardClosedCallback call is in-flight, and will call finishStop.\n\t\t_ = c.removeShardLocked(shardID, shard)\n\t}\n\n\tif err := c.ownership.verifyOwnership(shardID); err != nil {\n\t\treturn nil, err\n\t}\n\n\tif c.status.Load() == common.DaemonStatusStopped {\n\t\thostInfo := c.hostInfoProvider.HostInfo()\n\t\treturn nil, fmt.Errorf(\"ControllerImpl for host '%v' shutting down\", hostInfo.Identity())\n\t}\n\n\tshard, err := c.contextFactory.CreateContext(shardID, c.shardRemoveAndStop)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tc.historyShards[shardID] = shard\n\tmetrics.ShardContextCreatedCounter.With(c.taggedMetricsHandler).Record(1)\n\tc.contextTaggedLogger.Info(\"\", numShardsTag(len(c.historyShards)))\n\n\treturn shard, nil\n}\n\nfunc (c *ControllerImpl) removeShard(shardID int32, expected historyi.ControllableContext) historyi.ControllableContext {\n\tc.Lock()\n\tdefer c.Unlock()\n\treturn c.removeShardLocked(shardID, expected)\n}","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/shard/controller_impl.go#L238-L274","documentation":"The shard controller refuses to hand out new shard contexts once its daemon status is Stopped. GetShardByID -> getOrCreateShardContext checks status and fails fast with this error instead of creating shards on a shutting-down host. It signals the history service is shutting down and requests should be routed elsewhere.","triggerScenarios":"GetShardByID called on a controller whose c.status has been set to common.DaemonStatusStopped — i.e. during graceful shutdown while RPCs are still in flight, or a client pinned to a host that has just begun shutting down.","commonSituations":"Rolling deploys or host drains where the ring/ownership update lags behind the shutdown signal; retries hitting a recently stopped host; tests that stop the controller before finishing shard access.","solutions":["Retry the request against another host; the shard has likely been reassigned during shutdown","Verify the membership/ownership ring updates promptly on shutdown so clients stop routing to the draining host","If it happens outside shutdown, check for an accidental early Stop() of the controller or wrong daemon status handling"],"exampleFix":"// before\nshardCtx, err := controller.GetShardByID(ctx, shardID)\n// after\nshardCtx, err := controller.GetShardByID(ctx, shardID)\nif err != nil && strings.Contains(err.Error(), \"shutting down\") {\n\tshardCtx, err = controllerFor(shardID).GetShardByID(ctx, shardID) // retry other host\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"ctx, err := controller.GetShardByID(ctx, shardID)\nif err != nil && strings.Contains(err.Error(), \"shutting down\") {\n\t// do not retry same host; failover to another history host or re-resolve ownership\n}","preventionTips":["Route via ring/membership so stopped hosts are not selected","Drain traffic before setting DaemonStatusStopped","Treat this error as terminal for the host — never retry in place"],"tags":["go","history-service","shutdown","shard-controller"],"backgroundTag":"service-shutting-down","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}