What causes "failed to stream snapshot: %v" in hashicorp/nomad?

Re-run the snapshot command on a stable connection, ideally locally on the server

failed to stream snapshot: %v

Error message

failed to stream snapshot: %v

What it means

Wraps the error returned when the operator endpoint fails to copy the Raft snapshot bytes to the streaming RPC connection during an operator snapshot save. It indicates the snapshot was created but could not be transmitted to the client. The underlying io.Copy error is embedded via %v.

Source

Thrown at nomad/operator_endpoint.go:623

	op.srv.setQueryMeta(&reply.QueryMeta)

	// Take the snapshot and capture the index.
	snap, err := snapshot.New(op.logger.Named("snapshot"), op.srv.raft)
	reply.SnapshotChecksum = snap.Checksum()
	reply.Index = snap.Index()
	if err != nil {
		handleFailure(500, err)
		return
	}
	defer snap.Close()

	if err := encoder.Encode(&reply); err != nil {
		handleFailure(500, fmt.Errorf("failed to encode response: %v", err))
		return
	}
	if snap != nil {
		if _, err := io.Copy(conn, snap); err != nil {
			handleFailure(500, fmt.Errorf("failed to stream snapshot: %v", err))
		}
	}
}

func (op *Operator) snapshotRestore(conn io.ReadWriteCloser) {
	defer conn.Close()

	var args structs.SnapshotRestoreRequest
	var reply structs.SnapshotRestoreResponse
	decoder := codec.NewDecoder(conn, structs.MsgpackHandle)
	encoder := codec.NewEncoder(conn, structs.MsgpackHandle)

	handleFailure := func(code int, err error) {
		encoder.Encode(&structs.SnapshotRestoreResponse{
			ErrorCode: code,
			ErrorMsg:  err.Error(),
		})
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Re-run the snapshot command on a stable connection, ideally locally on the server
Check server logs for the wrapped underlying error to identify transport vs snapshot failure
Bypass proxies/LBs with short idle timeouts, or increase their stream timeout
Verify disk health on the leader if the snapshot reader itself fails

Example fix

// before
if _, err := io.Copy(conn, snap); err != nil {
    handleFailure(500, fmt.Errorf("failed to stream snapshot: %v", err))
}
// after
if _, err := io.Copy(conn, snap); err != nil {
    op.logger.Error("snapshot streaming failed", "error", err)
    handleFailure(500, fmt.Errorf("failed to stream snapshot: %w", err))
}

Defensive patterns

Strategy: retry

Validate before calling

// ensure connectivity to leader before snapshot
client, _ := api.NewClient(api.DefaultConfig())
leader, err := client.Status().Leader(nil)
if err != nil || leader == "" { return fmt.Errorf("no leader reachable") }

Try / catch

err := cmd.AgentClient.Operator().SnapshotSave(nil, writer)
if err != nil {
    if strings.Contains(err.Error(), "failed to stream snapshot") {
        // transient stream failure: retry with backoff
    }
}

Prevention

Run snapshot save locally on the server to avoid network cuts
Increase LB/proxy idle timeouts for long streams
Retry with exponential backoff on stream errors

When it happens

Trigger: Client disconnects mid-stream, network interruption during snapshot download, or the snapshot reader errors while io.Copy streams it over conn in snapshotSave.

Common situations: Operator runs 'nomad operator snapshot save' over a flaky VPN or through a load balancer with idle timeouts; connection is cut before the multi-GB snapshot finishes streaming.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0597f85163cc065a. Report an issue: GitHub.