weaviate/weaviate · error

get remote object: shard=%s: %w

Error message

get remote object: shard=%s: %w

What it means

Same read path as index.go:2451 but the remote branch: withShardOrRemote determined the shard is not local, and i.remote.GetObject failed to fetch the object from the responsible node. The error is wrapped with the shard name for diagnosis. Underlying causes are typically network failures between nodes or the remote node rejecting the request.

Source

Thrown at adapters/repos/db/index.go:2458

			obj, err = i.replicator.NodeObject(ctx, replProps.NodeName, shardName, id, props, addl)
		} else {
			obj, err = i.replicator.GetOne(ctx, routerTypes.ConsistencyLevel(replProps.ConsistencyLevel), shardName, id, props, addl)
		}
		return obj, err
	}

	err = i.withShardOrRemote(ctx, tenant, shardName, localShardOperationRead, 0,
		func(shard ShardLike) error {
			var err error
			if obj, err = shard.ObjectByID(ctx, id, props, addl); err != nil {
				return fmt.Errorf("get local object: shard=%s: %w", shardName, err)
			}
			return nil
		},
		func() error {
			var err error
			if obj, err = i.remote.GetObject(ctx, shardName, id, props, addl); err != nil {
				return fmt.Errorf("get remote object: shard=%s: %w", shardName, err)
			}
			return nil
		})
	return obj, err
}

func (i *Index) IncomingGetObject(ctx context.Context, shardName string,
	id strfmt.UUID, props search.SelectProperties,
	additional additional.Properties,
) (*storobj.Object, error) {
	shard, release, err := i.GetShard(ctx, shardName)
	if err != nil {
		return nil, err
	}
	defer release()

	if shard == nil {
		return nil, fmt.Errorf("local %s shard not found", shardName)

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check that all cluster nodes are up (GET /v1/nodes) and the target node's logs
2. Test inter-node connectivity (network policy, service DNS, ports 7001/8300-8302)
3. Retry the request — transient partitions and stale routing usually self-heal
4. Verify inter-node TLS certificates are valid and not expired
5. Repair the replica with async replication if the remote shard is persistently unhealthy

err := retry.Do(func() error {
  obj, getErr := client.Data().Getter().WithID(id).Do(ctx)
  if getErr != nil && strings.Contains(getErr.Error(), "get remote object") {
    return getErr // transient inter-node issue
  }
  return retry.Unrecoverable(getErr)
}, retry.Attempts(3), retry.BackOffDelay)

When it happens

Trigger: Get-by-ID on a cluster where the target shard's replica lives on another node; the remote RPC (i.remote.GetObject) fails due to network partition, the remote node being down/restarting, or TLS/auth misconfiguration between nodes.

Common situations: A node is down or OOM-killed mid-request; cluster network/DNS issues in Kubernetes; inter-node auth (certificate) expiry; shard migrated and routing cache momentarily stale.

Related errors

• failed to remotely execute read plan on replica %s at addr %
• updating schema
• connect: %w
• abort request: %w
• make request: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/ce30f4a94a4ededc. Report an issue: GitHub.