weaviate/weaviate · error

get shard %q: %w

Error message

get shard %q: %w

What it means

Index.createAsyncCheckpoint resolves the local shard via getLoadedShard to create an async-replication checkpoint. Any error from that lookup (other than the shard simply not being present) is wrapped as "get shard %q: %w". This is distinct from the benign not-hosted case, which returns nil so a class-wide fan-out can no-op safely.

Source

Thrown at adapters/repos/db/index_async_checkpoint.go:34

	"errors"
	"fmt"
	"sync"
	"sync/atomic"
	"time"

	"github.com/sirupsen/logrus"
	enterrors "github.com/weaviate/weaviate/entities/errors"
	"github.com/weaviate/weaviate/usecases/replica"
)

// createAsyncCheckpoint applies a fan-out create. createdAt must be the
// initiator's timestamp (convergence tie-breaker). Returns nil for shards
// not hosted on this node so a class-wide broadcast can no-op safely.
func (i *Index) createAsyncCheckpoint(ctx context.Context, shardName string, cutoffMs int64, createdAt time.Time) error {
	// Never load a shard from async replication.
	shard, release, err := i.getLoadedShard(shardName)
	if err != nil {
		return fmt.Errorf("get shard %q: %w", shardName, err)
	}
	if shard == nil {
		// Not hosted here is benign for a fan-out create; hosted-but-unloaded is a visible 412.
		if i.shards.Load(shardName) == nil {
			return nil
		}
		return fmt.Errorf("%w: shard %q not loaded on this node", errAsyncReplicationNotActive, shardName)
	}
	defer release()
	return shard.CreateAsyncCheckpoint(ctx, cutoffMs, createdAt)
}

// createAsyncCheckpointShards is best-effort: errors are joined (preserving
// errors.Is) so REST/gRPC mappers can still classify a sentinel result.
func (i *Index) createAsyncCheckpointShards(ctx context.Context, shardNames []string, cutoffMs int64, createdAt time.Time) error {
	var (
		errsMu sync.Mutex
		errs   []error

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Read the wrapped cause to identify why getLoadedShard failed and address it (e.g. wait for shard reload to finish).
2. Retry checkpoint creation after the shard settles (load/reload complete).
3. If the node is being drained, re-issue checkpoints once the node is stable.

err := idx.CreateAsyncCheckpoints(ctx, cutoff, shards)
if err != nil {
	if !errors.Is(err, errAsyncReplicationNotActive) {
		// get shard %q failure: retry after shard settles
	}
}

When it happens

Trigger: Calling CreateAsyncCheckpoints (or the internal createAsyncCheckpoint) when getLoadedShard fails for a shard: shard map access error, store-related failure while resolving the shard.

Common situations: Checkpoint creation racing with shard shutdown/reload; internal state errors in the shard registry on a node undergoing maintenance.

Related errors

• checkpoint createdAt is not newer than the active one
• checkpoint cutoff is not in this node's future
• %w: shard %q not loaded on this node
• cutoffMs must be > 0, got %d
• get local checkpoint status: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/6fc0c3071b0f6259. Report an issue: GitHub.