temporalio/temporal · error

ControllerImpl for host '%v' shutting down

Error message

ControllerImpl for host '%v' shutting down

What it means

The shard controller refuses to hand out new shard contexts once its daemon status is Stopped. GetShardByID -> getOrCreateShardContext checks status and fails fast with this error instead of creating shards on a shutting-down host. It signals the history service is shutting down and requests should be routed elsewhere.

Source

Thrown at service/history/shard/controller_impl.go:256

	// Check again with exclusive lock
	if shard, ok := c.historyShards[shardID]; ok {
		if shard.IsValid() {
			return shard, nil
		}

		// If the shard was invalid and still in the historyShards map, the
		// shardClosedCallback call is in-flight, and will call finishStop.
		_ = c.removeShardLocked(shardID, shard)
	}

	if err := c.ownership.verifyOwnership(shardID); err != nil {
		return nil, err
	}

	if c.status.Load() == common.DaemonStatusStopped {
		hostInfo := c.hostInfoProvider.HostInfo()
		return nil, fmt.Errorf("ControllerImpl for host '%v' shutting down", hostInfo.Identity())
	}

	shard, err := c.contextFactory.CreateContext(shardID, c.shardRemoveAndStop)
	if err != nil {
		return nil, err
	}
	c.historyShards[shardID] = shard
	metrics.ShardContextCreatedCounter.With(c.taggedMetricsHandler).Record(1)
	c.contextTaggedLogger.Info("", numShardsTag(len(c.historyShards)))

	return shard, nil
}

func (c *ControllerImpl) removeShard(shardID int32, expected historyi.ControllableContext) historyi.ControllableContext {
	c.Lock()
	defer c.Unlock()
	return c.removeShardLocked(shardID, expected)
}

View on GitHub (pinned to bde624efd1)

Solutions

1. Retry the request against another host; the shard has likely been reassigned during shutdown
2. Verify the membership/ownership ring updates promptly on shutdown so clients stop routing to the draining host
3. If it happens outside shutdown, check for an accidental early Stop() of the controller or wrong daemon status handling

Example fix

// before
shardCtx, err := controller.GetShardByID(ctx, shardID)
// after
shardCtx, err := controller.GetShardByID(ctx, shardID)
if err != nil && strings.Contains(err.Error(), "shutting down") {
	shardCtx, err = controllerFor(shardID).GetShardByID(ctx, shardID) // retry other host
}

ctx, err := controller.GetShardByID(ctx, shardID)
if err != nil && strings.Contains(err.Error(), "shutting down") {
	// do not retry same host; failover to another history host or re-resolve ownership
}

When it happens

Trigger: GetShardByID called on a controller whose c.status has been set to common.DaemonStatusStopped — i.e. during graceful shutdown while RPCs are still in flight, or a client pinned to a host that has just begun shutting down.

Common situations: Rolling deploys or host drains where the ring/ownership update lags behind the shutdown signal; retries hitting a recently stopped host; tests that stop the controller before finishing shard access.

Related errors

• unknown workflow state sync result: %d
• %w: zombie workflow cannot be updated
• ErrNonPositiveTotalNumShards
• Found key with non-zero pending task count but has no corres
• unknown task predicate type: %T

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/93c45334fe267855. Report an issue: GitHub.