VictoriaMetrics/VictoriaMetrics · error

cannot read %q at %s (remote path %q): %w

Error message

cannot read %q at %s (remote path %q): %w

What it means

The object was opened, but io.ReadAll on the response body failed while downloading the content. The object exists, yet its bytes could not be fully read (connection interrupted, checksum mismatch, decode error).

Source

Thrown at lib/backup/s3remote/s3.go:578

	}
	return true, nil
}

// ReadFile returns the content of filePath at fs.
func (fs *FS) ReadFile(filePath string) ([]byte, error) {
	p := path.Join(fs.Dir, filePath)
	input := &s3.GetObjectInput{
		Bucket: aws.String(fs.Bucket),
		Key:    aws.String(p),
	}
	o, err := fs.s3.GetObject(fs.ctx, input)
	if err != nil {
		return nil, fmt.Errorf("cannot open %q at %s (remote path %q): %w", filePath, fs, p, err)
	}
	defer o.Body.Close()
	b, err := io.ReadAll(o.Body)
	if err != nil {
		return nil, fmt.Errorf("cannot read %q at %s (remote path %q): %w", filePath, fs, p, err)
	}
	return b, nil
}

func (fs *FS) path(p common.Part) string {
	return p.RemotePath(fs.Dir)
}

type statReader struct {
	r    io.Reader
	size int64
}

func (sr *statReader) Read(p []byte) (int, error) {
	n, err := sr.r.Read(p)
	sr.size += int64(n)
	return n, err
}

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Retry the read; transient stream interruptions are common
2. Increase client read timeouts and disable aggressive idle cutoffs
3. Avoid proxies that buffer/kill long responses or use a stable VPC endpoint
4. If persistent, verify object integrity (ETag/checksum) — the stored object may be corrupt

// check object size before reading to anticipate long transfers
head, err := s3.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
if err != nil { return err }
// ensure client read timeout comfortably exceeds size/bandwidth

b, err := fs.ReadFile(filePath)
if err != nil && strings.Contains(err.Error(), "cannot read") {
	// transient stream failure: retry with backoff, verify ETag after
}

When it happens

Trigger: io.ReadAll(o.Body) returns non-nil mid-stream: connection reset while streaming a large part, TLS handshake failure mid-transfer, timeouts on slow links, or S3 returning a truncated body.

Common situations: Large backups over unstable networks; proxy/load-balancer idle timeouts killing long streams; clients with very short read timeouts.

Related errors

• cannot list dst parts: %w
• cannot upload %s to %s: %w
• cannot initialize connection to s3: %w
• cannot delete %q at %s: %w
• cannot list versions for %q at %s: %w

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/fcb1989837cca48f. Report an issue: GitHub.