kubernetes/kops · error

error reading %s: %v

Error message

error reading %s: %v

What it means

S3Path.WriteToWithContext streams the S3 object body into the caller's io.Writer via io.Copy. This error wraps any failure that occurs while reading the HTTP response body after the GetObject call itself succeeded — the object exists and was fetchable, but the stream broke mid-transfer. The original AWS SDK error (network reset, timeout, checksum mismatch, connection close) is embedded in the message.

Source

Thrown at util/pkg/vfs/s3fs.go:426

	klog.V(4).Infof("Reading file %q", p)

	request := &s3.GetObjectInput{}
	request.Bucket = aws.String(p.bucket)
	request.Key = aws.String(p.key)

	response, err := client.GetObject(ctx, request)
	if err != nil {
		if AWSErrorCode(err) == "NoSuchKey" {
			return 0, os.ErrNotExist
		}
		return 0, fmt.Errorf("error fetching %s: %v", p, err)
	}
	defer response.Body.Close()

	n, err := io.Copy(out, response.Body)
	if err != nil {
		return n, fmt.Errorf("error reading %s: %v", p, err)
	}
	return n, nil
}

func (p *S3Path) ReadDir() ([]Path, error) {
	ctx := context.TODO()
	client, err := p.client(ctx)
	if err != nil {
		return nil, err
	}

	prefix := p.key
	if prefix != "" && !strings.HasSuffix(prefix, "/") {
		prefix += "/"
	}
	request := &s3.ListObjectsV2Input{}
	request.Bucket = aws.String(p.bucket)
	request.Prefix = aws.String(prefix)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Retry the read — this is almost always a transient network failure; wrap ReadFile in a bounded retry with backoff.
2. Inspect the wrapped error in the message for the root cause (timeout vs reset vs checksum) and fix the network path (proxy, MTU, keepalive settings).
3. If it happens consistently on large objects, increase client timeouts or read in ranged GetObject chunks.
4. Verify VPC/DNS/firewall rules allow sustained connections to the S3 regional endpoint, not just the initial request.

Example fix

// before
stateStore, err := vfs.Context.ReadFile(ctx, path)
// after
var data []byte
for i := 0; i < 3; i++ {
    data, err = vfs.Context.ReadFile(ctx, path)
    if err == nil { break }
    time.Sleep(time.Duration(1<<i) * time.Second)
}

if _, err := s3Path.Path(); err == nil { /* path valid; body-read errors still possible, so preflight with a HEAD */ }
head, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
if err == nil && head.ContentLength != nil && *head.ContentLength > 5<<30 { // plan chunked/ranged reads for very large objects }

var data []byte
for attempt := 0; attempt < 4; attempt++ {
    data, err = vfs.Context.ReadFile(ctx, s3Path)
    if err == nil || errors.Is(err, os.ErrNotExist) { break }
    time.Sleep(time.Duration(250<<attempt) * time.Millisecond) // body reads are retryable
}

When it happens

Trigger: Calling ReadFile or WriteTo on an S3Path when the GetObject response body fails partway through io.Copy — e.g. TCP connection reset, TLS handshake/keepalive timeout, proxy dropping the connection, or an SDK response-body checksum validation failure on a large object.

Common situations: Reading large state files (cluster specs, etcd backups) over flaky links or through corporate proxies; long-running downloads where S3 closes idle connections; container/pod network interruptions in kOps controllers mid-sync.

Related errors

• reading primary MAC address from ec2 metadata: %w
• getting location for bucket %q: %w
• error querying ec2 metadata service (for region): %v
• failed to load AWS config: %w
• failed to get local-ipv4 address from ec2 metadata: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/a88a9bcf84c7b3cb. Report an issue: GitHub.