kubernetes/kops · error

error fetching %s: %v

Error message

error fetching %s: %v

What it means

WriteToWithContext downloads an object with GetObject. A NoSuchKey AWS error is translated to os.ErrNotExist; any other failure is wrapped as "error fetching <path>: <underlying error>". Note NoSuchBucket/AccessDenied are NOT mapped to ErrNotExist and surface here.

Source

Thrown at util/pkg/vfs/s3fs.go:420

// WriteToWithContext implements io.WriterTo, but adds a context
func (p *S3Path) WriteToWithContext(ctx context.Context, out io.Writer) (int64, error) {
	client, err := p.client(ctx)
	if err != nil {
		return 0, err
	}

	klog.V(4).Infof("Reading file %q", p)

	request := &s3.GetObjectInput{}
	request.Bucket = aws.String(p.bucket)
	request.Key = aws.String(p.key)

	response, err := client.GetObject(ctx, request)
	if err != nil {
		if AWSErrorCode(err) == "NoSuchKey" {
			return 0, os.ErrNotExist
		}
		return 0, fmt.Errorf("error fetching %s: %v", p, err)
	}
	defer response.Body.Close()

	n, err := io.Copy(out, response.Body)
	if err != nil {
		return n, fmt.Errorf("error reading %s: %v", p, err)
	}
	return n, nil
}

func (p *S3Path) ReadDir() ([]Path, error) {
	ctx := context.TODO()
	client, err := p.client(ctx)
	if err != nil {
		return nil, err
	}

	prefix := p.key

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped AWS error: NoSuchBucket → fix the bucket name in KOPS_STATE_STORE; AccessDenied → grant s3:GetObject
2. Check os.IsNotExist(err) first — missing keys are returned as os.ErrNotExist, not this error
3. Verify bucket region matches the client region (wrong region gives AuthorizationHeaderMalformed/301)
4. Ensure KMS key policy allows decryption if the bucket enforces SSE-KMS

Example fix

// before
data, err := vfs.Context.ReadFile(p)
if err != nil { return err } // conflates missing vs denied
// after
if _, err := vfs.Context.ReadFile(p); err != nil {
	if errors.Is(err, os.ErrNotExist) { return createDefaults() }
	return fmt.Errorf("reading state %s: %w", p, err)
}

// Probe bucket existence/permissions before reads
_, err := s3Client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: bucket})
if err != nil { return fmt.Errorf("state store bucket %s unusable: %w", bucket, err) }

data, err := vfs.Context.ReadFile(p)
if err != nil {
	if errors.Is(err, os.ErrNotExist) {
		return nil, errStateNotFound // missing key path
	}
	if code := AWSErrorCode(errors.Unwrap(err)); code == "NoSuchBucket" || code == "AccessDenied" {
		return nil, fmt.Errorf("state store misconfigured (%s): %w", code, err)
	}
	return nil, err
}

When it happens

Trigger: Calling ReadFile/WriteTo on an S3Path when GetObject fails for reasons other than missing key: AccessDenied on the key; NoSuchBucket (wrong bucket name/region); KMS decryption failure; throttling; network errors.

Common situations: Pointing KOPS_STATE_STORE at a nonexistent or mistyped bucket; credentials lacking s3:GetObject; a cluster state file deleted by another process but versioning absent so GetObject 404s (that path returns ErrNotExist, not this); VPC endpoint misconfigurations.

Related errors

• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w
• error removing %d files: %w
• error listing all versions of file %s: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/710132dfeabb73c5. Report an issue: GitHub.