weaviate/weaviate · error

snapshot change-log LSN on %s: %w

Error message

snapshot change-log LSN on %s: %w

What it means

SnapshotChangeLogLSN asks the source node, over gRPC, for the current change-log LSN snapshot and wraps any RPC error with this message. It means the source could not report the change-log position, so the caller cannot anchor the replication snapshot. The cause is preserved via %w.

Source

Thrown at cluster/replication/copier/copier_changelog.go:90

	}
	return changelogdrain.Drain(ctx, stream, apply)
}

// SnapshotChangeLogLSN returns the source's current change-log LSN without
// sealing it. The log stays writable; pair with a capped TailAndApply to
// drain a phase boundary without sealing.
func (c *Copier) SnapshotChangeLogLSN(ctx context.Context, srcNodeId, indexName, shardName, opID string) (uint64, error) {
	client, err := c.dialSource(ctx, srcNodeId)
	if err != nil {
		return 0, err
	}
	resp, err := client.SnapshotChangeLogLSN(ctx, &protocol.SnapshotChangeLogLSNRequest{
		IndexName: indexName,
		ShardName: shardName,
		OpId:      opID,
	})
	if err != nil {
		return 0, fmt.Errorf("snapshot change-log LSN on %s: %w", srcNodeId, err)
	}
	return resp.Lsn, nil
}

// FinalizeChangeLog seals the source's change-capture log and returns its
// final LSN. The caller need not compare to lastAppliedLSN — the server
// closes the stream with io.EOF once its tailer drains through finalLSN.
func (c *Copier) FinalizeChangeLog(ctx context.Context, srcNodeId, indexName, shardName, opID string) (uint64, error) {
	client, err := c.dialSource(ctx, srcNodeId)
	if err != nil {
		return 0, err
	}
	resp, err := client.FinalizeChangeLog(ctx, &protocol.FinalizeChangeLogRequest{
		IndexName: indexName,
		ShardName: shardName,
		OpId:      opID,
	})
	if err != nil {

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Inspect the wrapped cause for network vs protocol-level errors
2. Ensure StartChangeCapture succeeded for this opID before snapshotting
3. Verify source node health and gRPC connectivity (nodeSelector address/port)
4. Cancel and re-create the replication op if the source lost the op state
5. Check version compatibility of both nodes

Example fix

// before
lsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)
// after: guard against unregistered op
if _, ok := fsm.GetOpById(opID); !ok {
    return fmt.Errorf("op %d not registered; re-run StartChangeCapture", opID)
}
lsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)

// op must be registered and source reachable before snapshotting
if _, ok := fsm.GetOpById(opID); !ok {
    return fmt.Errorf("op %d not registered", opID)
}
if _, err := nodeSelector.NodeGRPCPort(srcNodeId); err != nil {
    return err
}

func sourceReady(sel NodeSelector, fsm *ReplicationFSM, srcNodeId string, opID uint64) bool {
    if _, err := sel.NodeGRPCPort(srcNodeId); err != nil { return false }
    _, ok := fsm.GetOpById(opID)
    return ok
}

lsn, err := copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)
if err != nil {
    if isUnreachable(err) {
        lsn, err = retryWithBackoff(ctx, func() (uint64, error) {
            return copier.SnapshotChangeLogLSN(ctx, index, shard, srcNode, opID)
        })
    }
    if err != nil { return fmt.Errorf("snapshot lsn failed: %v", err) }
}

When it happens

Trigger: Calling SnapshotChangeLogLSN(ctx, indexName, shardName, srcNodeId, opID) when the source node is unreachable, rejects the opID, the source shard is unloaded, or the change-capture log was not started/finalized on the source.

Common situations: opID never registered (StartChangeCapture skipped or failed); source node restarted and in-memory capture state lost; network partition; peer running an older Weaviate without SnapshotChangeLogLSN support.

Related errors

• finalize change log on %s: %w
• get gRPC connection to %s: %w
• marshal object: %w
• gRPC PutObject: %w
• gRPC PutObjects: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/27485846000777d7. Report an issue: GitHub.