kubernetes/kops · warning

Etag was not a valid MD5 sum: %q

Error message

Etag was not a valid MD5 sum: %q

What it means

S3Path.Hash uses the object's ETag as an MD5 hash for change detection. S3 ETags are only plain MD5s for simple (non-multipart) uploads; this error fires when the ETag cannot be hex-decoded to a 16-byte MD5 — typically because the object was written with multipart upload (ETag like "abc-2"), SSE-KMS/SSE-C encryption, or by a non-S3 backend (DigitalOcean Spaces via this same path).

Source

Thrown at util/pkg/vfs/s3fs.go:567

func (p *S3Path) PreferredHash() (*hashing.Hash, error) {
	return p.Hash(hashing.HashAlgorithmMD5)
}

func (p *S3Path) Hash(a hashing.HashAlgorithm) (*hashing.Hash, error) {
	if a != hashing.HashAlgorithmMD5 {
		return nil, nil
	}

	if p.etag == nil {
		return nil, nil
	}

	md5 := strings.Trim(*p.etag, "\"")

	md5Bytes, err := hex.DecodeString(md5)
	if err != nil {
		return nil, fmt.Errorf("Etag was not a valid MD5 sum: %q", *p.etag)
	}

	return &hashing.Hash{Algorithm: hashing.HashAlgorithmMD5, HashValue: md5Bytes}, nil
}

func (p *S3Path) GetHTTPsUrl(dualstack bool) (string, error) {
	ctx := context.TODO()

	bucketDetails, err := p.getBucketDetails(ctx)
	if err != nil {
		return "", fmt.Errorf("failed to get bucket details for %q: %w", p.String(), err)
	}

	resolver := s3.NewDefaultEndpointResolverV2()
	endpoint, err := resolver.ResolveEndpoint(ctx, s3.EndpointParameters{
		Bucket:       aws.String(bucketDetails.name),
		Region:       aws.String(bucketDetails.region),
		UseDualStack: aws.Bool(dualstack),

View on GitHub (pinned to 4c8573c808)

Solutions

1. Treat it as a non-fatal 'hash unavailable' signal — PreferredHash tolerates it; don't rely on ETag-based change detection for this object.
2. If MD5 matching matters, re-upload the object as a single-part upload without SSE-KMS so the ETag is a plain MD5.
3. For S3-compatible providers, configure the client to use the provider's own checksum mechanism instead of ETag comparison.
4. Check how the object was written (multipart threshold / encryption settings) and align the writer with what the reader expects.

Example fix

// before
etag := *headOutput.ETag // "d41d8...-2" (multipart)
// after
if !strings.Contains(*headOutput.ETag, "-") {
    md5 := strings.Trim(*headOutput.ETag, "\"")
    // safe to use as MD5
}

head, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
if err == nil && head.ETag != nil {
    etag := strings.Trim(*head.ETag, "\"")
    if strings.Contains(etag, "-") || len(etag) != 32 { // not a plain MD5 — skip ETag comparison } 
}

func isPlainMD5ETag(etag string) bool {
    e := strings.Trim(etag, "\"")
    if len(e) != 32 || strings.Contains(e, "-") { return false }
    _, err := hex.DecodeString(e)
    return err == nil
}

hash, err := s3Path.Hash()
if err != nil && strings.Contains(err.Error(), "Etag was not a valid MD5") {
    // fall back to size/mtime comparison or full-content compare
    return compareByFullDownload(ctx, s3Path)
}
if err != nil { return err }

When it happens

Trigger: Calling Hash or PreferredHash on an S3Path whose p.etag was set from a multipart-uploaded or encrypted object, so the ETag is not a 32-char hex MD5 (e.g. contains a '-N' suffix or is not hex at all).

Common situations: State files uploaded with aws s3 cp --sse aws:kms or via multipart; buckets on S3-compatible stores (DO Spaces, MinIO) where ETags aren't MD5; comparing kops state after an external tool rewrote the object.

Related errors

• failed to generate AWS IAM S3 access statements: %v
• cannot parse VFS path %q: %v
• path is not cluster readable: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/02d337a9184bdda6. Report an issue: GitHub.