{"record":{"id":"0597f85163cc065a","repo":"hashicorp/nomad","slug":"failed-to-stream-snapshot-v","errorCode":null,"errorMessage":"failed to stream snapshot: %v","messagePattern":"failed to stream snapshot: (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"nomad/operator_endpoint.go","lineNumber":623,"sourceCode":"\top.srv.setQueryMeta(&reply.QueryMeta)\n\n\t// Take the snapshot and capture the index.\n\tsnap, err := snapshot.New(op.logger.Named(\"snapshot\"), op.srv.raft)\n\treply.SnapshotChecksum = snap.Checksum()\n\treply.Index = snap.Index()\n\tif err != nil {\n\t\thandleFailure(500, err)\n\t\treturn\n\t}\n\tdefer snap.Close()\n\n\tif err := encoder.Encode(&reply); err != nil {\n\t\thandleFailure(500, fmt.Errorf(\"failed to encode response: %v\", err))\n\t\treturn\n\t}\n\tif snap != nil {\n\t\tif _, err := io.Copy(conn, snap); err != nil {\n\t\t\thandleFailure(500, fmt.Errorf(\"failed to stream snapshot: %v\", err))\n\t\t}\n\t}\n}\n\nfunc (op *Operator) snapshotRestore(conn io.ReadWriteCloser) {\n\tdefer conn.Close()\n\n\tvar args structs.SnapshotRestoreRequest\n\tvar reply structs.SnapshotRestoreResponse\n\tdecoder := codec.NewDecoder(conn, structs.MsgpackHandle)\n\tencoder := codec.NewEncoder(conn, structs.MsgpackHandle)\n\n\thandleFailure := func(code int, err error) {\n\t\tencoder.Encode(&structs.SnapshotRestoreResponse{\n\t\t\tErrorCode: code,\n\t\t\tErrorMsg:  err.Error(),\n\t\t})\n\t}","sourceCodeStart":605,"sourceCodeEnd":641,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/operator_endpoint.go#L605-L641","documentation":"Wraps the error returned when the operator endpoint fails to copy the Raft snapshot bytes to the streaming RPC connection during an operator snapshot save. It indicates the snapshot was created but could not be transmitted to the client. The underlying io.Copy error is embedded via %v.","triggerScenarios":"Client disconnects mid-stream, network interruption during snapshot download, or the snapshot reader errors while io.Copy streams it over conn in snapshotSave.","commonSituations":"Operator runs 'nomad operator snapshot save' over a flaky VPN or through a load balancer with idle timeouts; connection is cut before the multi-GB snapshot finishes streaming.","solutions":["Re-run the snapshot command on a stable connection, ideally locally on the server","Check server logs for the wrapped underlying error to identify transport vs snapshot failure","Bypass proxies/LBs with short idle timeouts, or increase their stream timeout","Verify disk health on the leader if the snapshot reader itself fails"],"exampleFix":"// before\nif _, err := io.Copy(conn, snap); err != nil {\n    handleFailure(500, fmt.Errorf(\"failed to stream snapshot: %v\", err))\n}\n// after\nif _, err := io.Copy(conn, snap); err != nil {\n    op.logger.Error(\"snapshot streaming failed\", \"error\", err)\n    handleFailure(500, fmt.Errorf(\"failed to stream snapshot: %w\", err))\n}","handlingStrategy":"retry","validationCode":"// ensure connectivity to leader before snapshot\nclient, _ := api.NewClient(api.DefaultConfig())\nleader, err := client.Status().Leader(nil)\nif err != nil || leader == \"\" { return fmt.Errorf(\"no leader reachable\") }","typeGuard":null,"tryCatchPattern":"err := cmd.AgentClient.Operator().SnapshotSave(nil, writer)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to stream snapshot\") {\n        // transient stream failure: retry with backoff\n    }\n}","preventionTips":["Run snapshot save locally on the server to avoid network cuts","Increase LB/proxy idle timeouts for long streams","Retry with exponential backoff on stream errors"],"tags":["network","io","snapshot","rpc"],"backgroundTag":"stream-interrupted","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}