{"record":{"id":"f451006d8e1d6cff","repo":"hashicorp/nomad","slug":"failed-to-read-snapshot-metadata-v","errorCode":null,"errorMessage":"failed to read snapshot metadata: %v","messagePattern":"failed to read snapshot metadata: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/snapshot/archive.go","lineNumber":209,"sourceCode":"\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed reading snapshot: %v\", err)\n\t\t}\n\n\t\tswitch hdr.Name {\n\t\tcase \"meta.json\":\n\t\t\t// Previously we used json.Decode to decode the archive stream. There are\n\t\t\t// edgecases in which it doesn't read all the bytes from the stream, even\n\t\t\t// though the json object is still being parsed properly. Since we\n\t\t\t// simultaneously feeded everything to metaHash, our hash ended up being\n\t\t\t// different than what we calculated when creating the snapshot. Which in\n\t\t\t// turn made the snapshot verification fail. By explicitly reading the\n\t\t\t// whole thing first we ensure that we calculate the correct hash\n\t\t\t// independent of how json.Decode works internally.\n\t\t\tbuf, err := io.ReadAll(io.TeeReader(archive, metaHash))\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read snapshot metadata: %v\", err)\n\t\t\t}\n\t\t\tif err := json.Unmarshal(buf, &metadata); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to decode snapshot metadata: %v\", err)\n\t\t\t}\n\n\t\tcase \"state.bin\":\n\t\t\tif _, err := io.Copy(io.MultiWriter(snap, snapHash), archive); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read or write snapshot data: %v\", err)\n\t\t\t}\n\n\t\tcase \"SHA256SUMS\":\n\t\t\tif _, err := io.Copy(&shaBuffer, archive); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read snapshot hashes: %v\", err)\n\t\t\t}\n\n\t\tdefault:\n\t\t\treturn fmt.Errorf(\"unexpected file %q in snapshot\", hdr.Name)\n\t\t}","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/snapshot/archive.go#L191-L227","documentation":"This error wraps an io.ReadAll failure while reading the meta.json entry (Raft snapshot metadata) from the tar archive, with bytes teed into the SHA-256 hash. It indicates the underlying snapshot stream failed mid-read of the metadata file (I/O error on the source reader), not a JSON problem. The wrapped %v contains the underlying read error.","triggerScenarios":"read() hits the 'meta.json' entry and io.ReadAll(io.TeeReader(archive, metaHash)) errors: the input stream (file, network connection, HTTP body) fails or is reset partway through reading the metadata section of the tar.","commonSituations":"Network connection reset while downloading a snapshot from a remote agent; snapshot file on flaky storage; TLS/SSH tunnel dropping mid-transfer; reading a snapshot mounted over a failing network share.","solutions":["Retry the snapshot download/restore operation; transient stream failures are the usual cause","Verify stability of the storage/network path hosting the snapshot file","Re-take the snapshot if the source copy is consistently unreadable","Check the wrapped error for the concrete I/O failure (connection reset, permission, device error) and address it directly"],"exampleFix":"// before\nerr := snapshot.Restore(logOut, snap)\n// after\nvar lastErr error\nfor i := 0; i < 3; i++ {\n    if err := snapshot.Restore(logOut, openSnapshot()); err != nil {\n        lastErr = err\n        time.Sleep(time.Second)\n        continue\n    }\n    lastErr = nil\n    break\n}\nreturn lastErr","handlingStrategy":"retry","validationCode":"// pre-check: source is reachable and stable before starting\nf, err := os.Open(snapshotPath)\nif err != nil {\n    return fmt.Errorf(\"snapshot source unavailable: %w\", err)\n}\nif _, err := f.Stat(); err != nil {\n    return fmt.Errorf(\"snapshot source unreadable: %w\", err)\n}","typeGuard":"func isMetadataReadErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to read snapshot metadata\")\n}","tryCatchPattern":"var err error\nfor attempt := 0; attempt < 3; attempt++ {\n    err = snapshot.Restore(logOut, reopenSnapshot())\n    if err == nil || !strings.Contains(err.Error(), \"failed to read snapshot metadata\") {\n        break\n    }\n    time.Sleep(2 * time.Second) // transient stream failure: back off and retry\n}\nreturn err","preventionTips":["Use retry with backoff for snapshot downloads over the network","Avoid reading snapshots from unreliable network mounts; copy locally first","Monitor connection stability (resets, TLS errors) on snapshot transfer paths","Re-verify snapshots after any transfer with 'consul snapshot inspect'"],"tags":["go","io","snapshot","network"],"backgroundTag":"stream-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}