weaviate/weaviate · error

init shard %q metrics: %w

Error message

init shard %q metrics: %w

What it means

Weaviate wraps the failure of NewMetrics() during shard creation in NewShard (adapters/repos/db/shard_init.go:66). NewMetrics registers shard-level Prometheus metrics (counters/gauges/histograms labeled by class and shard) with the provided registerer. If registration fails — typically because a metric with the same fully-qualified name is already registered — shard startup aborts and the shard is not created.

Source

Thrown at adapters/repos/db/shard_init.go:66

		"shard":  shardName,
		"index":  index.ID(),
	}).Debugf("initializing shard %q", shardName)

	if err := shardusage.RemoveComputedUsageDataForUnloadedShard(index.path(), shardName); err != nil {
		return nil, fmt.Errorf("shard %q: remove computed usage file for unloaded shard: %w", shardName, err)
	}

	if err := newPropertyDeleteIndexHelper().ensureBucketsAreRemovedForNonExistentPropertyIndexes(index.path(), shardName, class); err != nil {
		return nil, fmt.Errorf("shard %q: remove nonexistent property index buckets: %w", shardName, err)
	}

	if err := newVectorDropIndexHelper().ensureFilesAreRemovedForDroppedVectorIndexes(index.path(), shardName, class); err != nil {
		return nil, fmt.Errorf("shard %q: remove dropped vector index files: %w", shardName, err)
	}

	metrics, err := NewMetrics(index.logger, promMetrics, string(index.Config.ClassName), shardName)
	if err != nil {
		return nil, fmt.Errorf("init shard %q metrics: %w", shardName, err)
	}
	if index.Config.LazySegmentsDisabled {
		lazyLoadSegments = false // disabled globally
	}

	shutCtx, shutCtxCancel := context.WithCancelCause(context.Background())

	s := &Shard{
		index:       index,
		class:       class,
		name:        shardName,
		promMetrics: promMetrics,
		metrics:     metrics,
		slowQueryReporter: helpers.NewSlowQueryReporter(index.Config.QuerySlowLogEnabled,
			index.Config.QuerySlowLogThreshold, index.logger),
		replicationMap:   pendingReplicaTasks{Tasks: make(map[string]replicaTask, 32)},
		centralJobQueue:  jobQueueCh,
		scheduler:        scheduler,

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check the wrapped error for prometheus.AlreadyRegisteredError and ensure the previous shard's metrics are unregistered before re-creating it (complete offload/cleanup before onload)
2. Verify class and shard names don't collide after metric-name sanitization; rename the offending shard or class
3. If embedding Weaviate or running tests, pass a fresh prometheus.NewRegistry() per shard instead of reusing one registerer
4. Check disk/permissions are not the cause — inspect the full wrapped chain for the underlying NewMetrics error

Example fix

// before (test): reuse one registry for two shards
reg := prometheus.NewRegistry()
s1, _ := db.NewShard(ctx, monitoring.NewPrometheusMetrics(reg), "shard-1", idx, class, ...)
s2, _ := db.NewShard(ctx, monitoring.NewPrometheusMetrics(reg), "shard-1", idx2, class, ...) // duplicate registration
// after
s2, _ := db.NewShard(ctx, monitoring.NewPrometheusMetrics(prometheus.NewRegistry()), "shard-1", idx2, class, ...)

// Before creating a shard with a custom registerer, ensure no metric of the same
// name is registered. In tests/embedding scenarios:
func canRegister(reg prometheus.Registerer, name string) bool {
	collector := prometheus.NewCounter(prometheus.CounterOpts{Name: name})
	if err := reg.Register(collector); err != nil {
		reg.Unregister(collector) // roll back the probe
		return false
	}
	reg.Unregister(collector)
	return true
}

shard, err := db.NewShard(ctx, promMetrics, shardName, index, class, ...)
if err != nil {
	var are prometheus.AlreadyRegisteredError
	if errors.As(err, &are) {
		// duplicate metric registration: clean up old registration and retry once
	}
	return fmt.Errorf("shard creation failed: %w", err)
}

When it happens

Trigger: NewShard is called and NewMetrics returns an error, most commonly prometheus.AlreadyRegisteredError from duplicate metric registration: creating two shards whose metric name components (class name/shard name) collide after sanitization, re-registering on shard re-creation (e.g. tenant offloading/re-onloading) without an unregister, or passing an already-populated registerer.

Common situations: Operators see this when starting Weaviate with a PROMETHEUS_MONITORING_ENABLED setup where a previous registration was not cleaned up, when class/shard names sanitize to the same Prometheus label-safe string, or in tests that reuse a single prometheus.Registerer across multiple NewShard calls. Also possible after offload/onload cycles if cleanup of the old shard's metrics failed.

Related errors

• create metrics for index %q: %w
• init lsmkv metrics: %w
• shard metrics: %w
• metric %s already registered but not as a CounterVec
• metric %s already registered but not as a Counter

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/118fd4c5314dc59b. Report an issue: GitHub.