{"record":{"id":"ce30f4a94a4ededc","repo":"weaviate/weaviate","slug":"get-remote-object-shard-s-w","errorCode":null,"errorMessage":"get remote object: shard=%s: %w","messagePattern":"get remote object: shard=(.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/index.go","lineNumber":2458,"sourceCode":"\t\t\tobj, err = i.replicator.NodeObject(ctx, replProps.NodeName, shardName, id, props, addl)\n\t\t} else {\n\t\t\tobj, err = i.replicator.GetOne(ctx, routerTypes.ConsistencyLevel(replProps.ConsistencyLevel), shardName, id, props, addl)\n\t\t}\n\t\treturn obj, err\n\t}\n\n\terr = i.withShardOrRemote(ctx, tenant, shardName, localShardOperationRead, 0,\n\t\tfunc(shard ShardLike) error {\n\t\t\tvar err error\n\t\t\tif obj, err = shard.ObjectByID(ctx, id, props, addl); err != nil {\n\t\t\t\treturn fmt.Errorf(\"get local object: shard=%s: %w\", shardName, err)\n\t\t\t}\n\t\t\treturn nil\n\t\t},\n\t\tfunc() error {\n\t\t\tvar err error\n\t\t\tif obj, err = i.remote.GetObject(ctx, shardName, id, props, addl); err != nil {\n\t\t\t\treturn fmt.Errorf(\"get remote object: shard=%s: %w\", shardName, err)\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\treturn obj, err\n}\n\nfunc (i *Index) IncomingGetObject(ctx context.Context, shardName string,\n\tid strfmt.UUID, props search.SelectProperties,\n\tadditional additional.Properties,\n) (*storobj.Object, error) {\n\tshard, release, err := i.GetShard(ctx, shardName)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer release()\n\n\tif shard == nil {\n\t\treturn nil, fmt.Errorf(\"local %s shard not found\", shardName)","sourceCodeStart":2440,"sourceCodeEnd":2476,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/index.go#L2440-L2476","documentation":"Same read path as index.go:2451 but the remote branch: withShardOrRemote determined the shard is not local, and i.remote.GetObject failed to fetch the object from the responsible node. The error is wrapped with the shard name for diagnosis. Underlying causes are typically network failures between nodes or the remote node rejecting the request.","triggerScenarios":"Get-by-ID on a cluster where the target shard's replica lives on another node; the remote RPC (i.remote.GetObject) fails due to network partition, the remote node being down/restarting, or TLS/auth misconfiguration between nodes.","commonSituations":"A node is down or OOM-killed mid-request; cluster network/DNS issues in Kubernetes; inter-node auth (certificate) expiry; shard migrated and routing cache momentarily stale.","solutions":["Check that all cluster nodes are up (GET /v1/nodes) and the target node's logs","Test inter-node connectivity (network policy, service DNS, ports 7001/8300-8302)","Retry the request — transient partitions and stale routing usually self-heal","Verify inter-node TLS certificates are valid and not expired","Repair the replica with async replication if the remote shard is persistently unhealthy"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := retry.Do(func() error {\n  obj, getErr := client.Data().Getter().WithID(id).Do(ctx)\n  if getErr != nil && strings.Contains(getErr.Error(), \"get remote object\") {\n    return getErr // transient inter-node issue\n  }\n  return retry.Unrecoverable(getErr)\n}, retry.Attempts(3), retry.BackOffDelay)","preventionTips":["Monitor node liveness (GET /v1/nodes) and alert on down nodes","Secure and rotate inter-node TLS certificates before expiry","Avoid aggressive per-request timeouts on clustered reads","Keep network policies open for Weaviate peer ports"],"tags":["go","network","cluster","remote-read"],"backgroundTag":"inter-node-request-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}