hashicorp/nomad · error

failed to read after %v: %v

Error message

failed to read after %v: %v

What it means

Restore wraps the incoming reader so every read error after io.EOF's byte count is annotated with the total bytes read so far. Any non-EOF read error while streaming the snapshot into the restorer is reported as 'failed to read after N bytes: <err>', which pinpoints how far into the snapshot the failure occurred.

Source

Thrown at helper/snapshot/snapshot.go:246

	extra, err := io.ReadAll(decomp) // ReadAll consumes the EOF
	if err != nil {
		return err
	} else if len(extra) != 0 {
		return fmt.Errorf("%d unread uncompressed bytes remain", len(extra))
	}
	return nil
}

type readWrapper struct {
	in io.Reader
	c  int
}

func (r *readWrapper) Read(b []byte) (int, error) {
	n, err := r.in.Read(b)
	r.c += n
	if err != nil && err != io.EOF {
		return n, fmt.Errorf("failed to read after %v: %v", r.c, err)
	}
	return n, err
}

// Restore takes the snapshot from the reader and attempts to apply it to the
// given Raft instance.
func Restore(logger hclog.Logger, in io.Reader, r *raft.Raft) error {
	// Wrap the reader in a gzip decompressor.
	decomp, err := gzip.NewReader(&readWrapper{in, 0})
	if err != nil {
		return fmt.Errorf("failed to decompress snapshot: %v", err)
	}
	defer func() {
		if err := decomp.Close(); err != nil {
			logger.Error("Failed to close snapshot decompressor", "error", err)
		}
	}()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Use the N-bytes position to compare against the expected snapshot size and confirm where truncation happened.
2. Retry the restore with a fresh download; transient network errors usually clear.
3. Ensure the source (HTTP client/CLI) uses streaming with adequate timeouts and connection keep-alive for large snapshots.
4. Verify the local snapshot file's integrity (checksum) if reading from disk — failing storage is the likely cause otherwise.

// check expected size is available before restore
func expectSize(rc io.ReadCloser, want int64) io.ReadCloser { return sizeCheck{rc, want, 0} }
// abort early if the stream ends well before the expected snapshot size

err := snapshot.Restore(logger, in, r)
if err != nil && strings.Contains(err.Error(), "failed to read after") {
	// transient network errors: safe to retry — state untouched if read failed
	return retryWithBackoff(func() error { return redownloadAndRestore() })
}

When it happens

Trigger: Restore (via snapshotRestore) where r.in.Read returns a non-EOF error mid-stream: network connection reset while downloading the snapshot, TLS errors, storage read failures on a local snapshot file, or timeouts closing the body.

Common situations: Long snapshots over flaky connections hit read timeouts; proxy closes idle connections mid-transfer; snapshot file on degraded storage returning EIO partway through.

Related errors

• failed to read snapshot metadata: %v
• failed to read or write snapshot data: %v
• failed to stream snapshot: %v
• failed to read snapshot: %w
• failed to read snapshot: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b29ab4752a6bf490. Report an issue: GitHub.