weaviate/weaviate · error

local shard %q: %w

Error message

local shard %q: %w

What it means

getOrInitShard (or similar shard lookup) was called after the Index was shut down; it wraps errAlreadyShutdown. The index-level closed flag under closeLock indicates the whole collection index has been closed, so no shard can be served.

Source

Thrown at adapters/repos/db/index.go:3575

) {
	return i.getOptInitLocalShard(ctx, shardName, true, callerMovement)
}

// getLoadedShard returns the shard only if it is already loaded, never
// materializing one, plus a release that drops the shutdown refcount. A nil
// shard and a nil error mean the shard is not loaded here.
//
// The refcount rather than shardCreateLocks keeps the shard alive, because
// shardCreateLocks is on the shard's own read and write path and callers can
// run long. Same reasoning as backupShardWithHardlinks.
func (i *Index) getLoadedShard(shardName string) (shard ShardLike, release func(), err error) {
	// Every shard teardown holds one of these two: Index.Shutdown takes closeLock
	// for write, the unload paths take shardCreateLocks for write.
	i.closeLock.RLock()
	defer i.closeLock.RUnlock()

	if i.closed {
		return nil, func() {}, fmt.Errorf("local shard %q: %w", shardName, errAlreadyShutdown)
	}

	i.shardCreateLocks.RLock(shardName)
	defer i.shardCreateLocks.RUnlock(shardName)

	shard = i.shards.Loaded(shardName)
	if shard == nil {
		return nil, func() {}, nil
	}
	// Both locks span the map read and the refcount: a lazy shard's
	// preventShutdown loads, so a teardown landing in between would have it
	// rebuild the shard outside i.shards, where nothing can ever shut it down.
	release, err = shard.preventShutdown()
	if err != nil {
		return nil, func() {}, fmt.Errorf("shard %q, no shutdown: %w", shardName, err)
	}
	return shard, release, nil
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Retry against another node — this node is shutting down; load balancers should mark it unhealthy.
2. Check errors.Is(err, errAlreadyShutdown) to distinguish from other local-shard failures.
3. Fix client routing so drained nodes stop receiving traffic during rolling upgrades.
4. If this happens outside a shutdown, investigate what called Index.Shutdown (schema deletion of the class, node drop).

Example fix

shard, release, err := i.getOrInitShard(ctx, shardName)
if err != nil {
    if errors.Is(err, errAlreadyShutdown) {
        return ErrNodeDraining // surface as retryable/redirect
    }
    return err
}

// health-check the node before sending traffic
health, _ := client.Nodes().Getter().Do(ctx) // node status should be HEALTHY

if errors.Is(err, errAlreadyShutdown) {
    return redirectOrRetry(otherNode) // node is draining
}

When it happens

Trigger: Any object read/write/lookup arriving on an index whose Shutdown() already completed — e.g. requests in flight during graceful node shutdown, or use of a DB/Index handle after Close.

Common situations: Client requests hitting a node mid-rolling-restart; load balancer not yet draining the node; application code holding a client connection to a node that has been stopped.

Related errors

• reactivate shard %q: %w
• already shut or dropped
• shard shutdown in progress
• shard still in use
• previous shutdown attempt failed mid-teardown

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/8c11e7f7b6459807. Report an issue: GitHub.