{"record":{"id":"27485846000777d7","repo":"weaviate/weaviate","slug":"snapshot-change-log-lsn-on-s-w","errorCode":null,"errorMessage":"snapshot change-log LSN on %s: %w","messagePattern":"snapshot change-log LSN on (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cluster/replication/copier/copier_changelog.go","lineNumber":90,"sourceCode":"\t}\n\treturn changelogdrain.Drain(ctx, stream, apply)\n}\n\n// SnapshotChangeLogLSN returns the source's current change-log LSN without\n// sealing it. The log stays writable; pair with a capped TailAndApply to\n// drain a phase boundary without sealing.\nfunc (c *Copier) SnapshotChangeLogLSN(ctx context.Context, srcNodeId, indexName, shardName, opID string) (uint64, error) {\n\tclient, err := c.dialSource(ctx, srcNodeId)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\tresp, err := client.SnapshotChangeLogLSN(ctx, &protocol.SnapshotChangeLogLSNRequest{\n\t\tIndexName: indexName,\n\t\tShardName: shardName,\n\t\tOpId:      opID,\n\t})\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"snapshot change-log LSN on %s: %w\", srcNodeId, err)\n\t}\n\treturn resp.Lsn, nil\n}\n\n// FinalizeChangeLog seals the source's change-capture log and returns its\n// final LSN. The caller need not compare to lastAppliedLSN — the server\n// closes the stream with io.EOF once its tailer drains through finalLSN.\nfunc (c *Copier) FinalizeChangeLog(ctx context.Context, srcNodeId, indexName, shardName, opID string) (uint64, error) {\n\tclient, err := c.dialSource(ctx, srcNodeId)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\tresp, err := client.FinalizeChangeLog(ctx, &protocol.FinalizeChangeLogRequest{\n\t\tIndexName: indexName,\n\t\tShardName: shardName,\n\t\tOpId:      opID,\n\t})\n\tif err != nil {","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/cluster/replication/copier/copier_changelog.go#L72-L108","documentation":"SnapshotChangeLogLSN asks the source node, over gRPC, for the current change-log LSN snapshot and wraps any RPC error with this message. It means the source could not report the change-log position, so the caller cannot anchor the replication snapshot. The cause is preserved via %w.","triggerScenarios":"Calling SnapshotChangeLogLSN(ctx, indexName, shardName, srcNodeId, opID) when the source node is unreachable, rejects the opID, the source shard is unloaded, or the change-capture log was not started/finalized on the source.","commonSituations":"opID never registered (StartChangeCapture skipped or failed); source node restarted and in-memory capture state lost; network partition; peer running an older Weaviate without SnapshotChangeLogLSN support.","solutions":["Inspect the wrapped cause for network vs protocol-level errors","Ensure StartChangeCapture succeeded for this opID before snapshotting","Verify source node health and gRPC connectivity (nodeSelector address/port)","Cancel and re-create the replication op if the source lost the op state","Check version compatibility of both nodes"],"exampleFix":"// before\nlsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)\n// after: guard against unregistered op\nif _, ok := fsm.GetOpById(opID); !ok {\n    return fmt.Errorf(\"op %d not registered; re-run StartChangeCapture\", opID)\n}\nlsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)","handlingStrategy":"retry","validationCode":"// op must be registered and source reachable before snapshotting\nif _, ok := fsm.GetOpById(opID); !ok {\n    return fmt.Errorf(\"op %d not registered\", opID)\n}\nif _, err := nodeSelector.NodeGRPCPort(srcNodeId); err != nil {\n    return err\n}","typeGuard":"func sourceReady(sel NodeSelector, fsm *ReplicationFSM, srcNodeId string, opID uint64) bool {\n    if _, err := sel.NodeGRPCPort(srcNodeId); err != nil { return false }\n    _, ok := fsm.GetOpById(opID)\n    return ok\n}","tryCatchPattern":"lsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)\nif err != nil {\n    if isUnreachable(err) {\n        lsn, err = retryWithBackoff(ctx, func() (uint64, error) {\n            return copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)\n        })\n    }\n    if err != nil { return fmt.Errorf(\"snapshot lsn failed: %v\", err) }\n}","preventionTips":["Always call StartChangeCapture and confirm success before snapshotting","Verify op state in the FSM before RPCs","Keep node versions aligned across the cluster"],"tags":["grpc","replication","lsn"],"backgroundTag":"grpc-rpc-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}