{"record":{"id":"02d337a9184bdda6","repo":"kubernetes/kops","slug":"etag-was-not-a-valid-md5-sum-q-02d337","errorCode":null,"errorMessage":"Etag was not a valid MD5 sum: %q","messagePattern":"Etag was not a valid MD5 sum: %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"util/pkg/vfs/s3fs.go","lineNumber":567,"sourceCode":"\nfunc (p *S3Path) PreferredHash() (*hashing.Hash, error) {\n\treturn p.Hash(hashing.HashAlgorithmMD5)\n}\n\nfunc (p *S3Path) Hash(a hashing.HashAlgorithm) (*hashing.Hash, error) {\n\tif a != hashing.HashAlgorithmMD5 {\n\t\treturn nil, nil\n\t}\n\n\tif p.etag == nil {\n\t\treturn nil, nil\n\t}\n\n\tmd5 := strings.Trim(*p.etag, \"\\\"\")\n\n\tmd5Bytes, err := hex.DecodeString(md5)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"Etag was not a valid MD5 sum: %q\", *p.etag)\n\t}\n\n\treturn &hashing.Hash{Algorithm: hashing.HashAlgorithmMD5, HashValue: md5Bytes}, nil\n}\n\nfunc (p *S3Path) GetHTTPsUrl(dualstack bool) (string, error) {\n\tctx := context.TODO()\n\n\tbucketDetails, err := p.getBucketDetails(ctx)\n\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to get bucket details for %q: %w\", p.String(), err)\n\t}\n\n\tresolver := s3.NewDefaultEndpointResolverV2()\n\tendpoint, err := resolver.ResolveEndpoint(ctx, s3.EndpointParameters{\n\t\tBucket:       aws.String(bucketDetails.name),\n\t\tRegion:       aws.String(bucketDetails.region),\n\t\tUseDualStack: aws.Bool(dualstack),","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/s3fs.go#L549-L585","documentation":"S3Path.Hash uses the object's ETag as an MD5 hash for change detection. S3 ETags are only plain MD5s for simple (non-multipart) uploads; this error fires when the ETag cannot be hex-decoded to a 16-byte MD5 — typically because the object was written with multipart upload (ETag like \"abc-2\"), SSE-KMS/SSE-C encryption, or by a non-S3 backend (DigitalOcean Spaces via this same path).","triggerScenarios":"Calling Hash or PreferredHash on an S3Path whose p.etag was set from a multipart-uploaded or encrypted object, so the ETag is not a 32-char hex MD5 (e.g. contains a '-N' suffix or is not hex at all).","commonSituations":"State files uploaded with aws s3 cp --sse aws:kms or via multipart; buckets on S3-compatible stores (DO Spaces, MinIO) where ETags aren't MD5; comparing kops state after an external tool rewrote the object.","solutions":["Treat it as a non-fatal 'hash unavailable' signal — PreferredHash tolerates it; don't rely on ETag-based change detection for this object.","If MD5 matching matters, re-upload the object as a single-part upload without SSE-KMS so the ETag is a plain MD5.","For S3-compatible providers, configure the client to use the provider's own checksum mechanism instead of ETag comparison.","Check how the object was written (multipart threshold / encryption settings) and align the writer with what the reader expects."],"exampleFix":"// before\netag := *headOutput.ETag // \"d41d8...-2\" (multipart)\n// after\nif !strings.Contains(*headOutput.ETag, \"-\") {\n    md5 := strings.Trim(*headOutput.ETag, \"\\\"\")\n    // safe to use as MD5\n}","handlingStrategy":"type-guard","validationCode":"head, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})\nif err == nil && head.ETag != nil {\n    etag := strings.Trim(*head.ETag, \"\\\"\")\n    if strings.Contains(etag, \"-\") || len(etag) != 32 { // not a plain MD5 — skip ETag comparison } \n}","typeGuard":"func isPlainMD5ETag(etag string) bool {\n    e := strings.Trim(etag, \"\\\"\")\n    if len(e) != 32 || strings.Contains(e, \"-\") { return false }\n    _, err := hex.DecodeString(e)\n    return err == nil\n}","tryCatchPattern":"hash, err := s3Path.Hash()\nif err != nil && strings.Contains(err.Error(), \"Etag was not a valid MD5\") {\n    // fall back to size/mtime comparison or full-content compare\n    return compareByFullDownload(ctx, s3Path)\n}\nif err != nil { return err }","preventionTips":["Don't rely on ETag equality for multipart or SSE-KMS encrypted objects.","Upload state files single-part and unencrypted (or use your own checksums) if hash comparison matters.","Guard with a '-'-suffix check on the ETag before treating it as MD5.","For S3-compatible providers (MinIO/DO Spaces), use provider-specific integrity metadata."],"tags":["aws","s3","md5","etag","hashing"],"backgroundTag":"s3-etag-not-md5","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}