hashicorp/terraform · critical

failed to open file at %v: checksum mismatch, %s != %s

Error message

failed to open file at %v: checksum mismatch, %s != %s

What it means

Raised by getObject() when the MD5 computed over the downloaded bytes does not equal the 'X-Cos-Meta-Md5' stored when the object was written. This is an active integrity check: the object's content was altered on the wire or in storage since it was uploaded. The backend treats any mismatch as corrupt state and refuses to load it.

Source

Thrown at internal/backend/remote-state/cos/client.go:219

	checksum = rsp.Header.Get("X-Cos-Meta-Md5")
	log.Printf("[DEBUG] getObject %s: checksum: %s", cosFile, checksum)
	if len(checksum) != 32 {
		err = fmt.Errorf("failed to open file at %v: checksum %s invalid", cosFile, checksum)
		return
	}

	exists = true
	data, err = ioutil.ReadAll(rsp.Body)
	log.Printf("[DEBUG] getObject %s: data length: %d", cosFile, len(data))
	if err != nil {
		err = fmt.Errorf("failed to open file at %v: %v", cosFile, err)
		return
	}

	check := fmt.Sprintf("%x", md5.Sum(data))
	log.Printf("[DEBUG] getObject %s: check: %s", cosFile, check)
	if check != checksum {
		err = fmt.Errorf("failed to open file at %v: checksum mismatch, %s != %s", cosFile, check, checksum)
		return
	}

	return
}

// putObject put object to remote
func (c *remoteClient) putObject(cosFile string, data []byte) error {
	opt := &cos.ObjectPutOptions{
		ObjectPutHeaderOptions: &cos.ObjectPutHeaderOptions{
			XCosMetaXXX: &http.Header{
				"X-Cos-Meta-Md5": []string{fmt.Sprintf("%x", md5.Sum(data))},
			},
		},
		ACLHeaderOptions: &cos.ACLHeaderOptions{
			XCosACL: c.acl,
		},
	}

View on GitHub (pinned to c9def3e214)

Solutions

1. Restore from the last known-good state backup ('terraform state push' a local copy or a prior version from COS versioning/bucket backup).
2. Verify no concurrent Terraform runs or external writers are targeting the same key; enable state locking and ensure the lock tag is functional.
3. Compare the downloaded bytes against a known-good copy to confirm corruption vs. a legitimately changed object, then re-push the correct state.
4. If a proxy/CDN is in the path, bypass it or fix content-encoding handling so the body is delivered byte-for-byte.

Example fix

# recover from a corrupt state object
terraform state pull > /tmp/check.tfstate.json   # likely fails with checksum mismatch
# restore a known-good copy and re-push
md5sum /tmp/good.tfstate.json
terraform state push --force /tmp/good.tfstate.json

// After download, recompute md5 and compare before using the bytes.
func verifyStateIntegrity(data []byte, checksum string) error {
    got := fmt.Sprintf("%x", md5.Sum(data))
    if len(checksum) != 32 {
        return fmt.Errorf("missing/invalid stored checksum")
    }
    if got != checksum {
        return fmt.Errorf("checksum mismatch %s != %s", got, checksum)
    }
    return nil
}

if _, _, _, err := c.getObject(c.stateFile); err != nil {
    if strings.Contains(err.Error(), "checksum mismatch") {
        // DO NOT overwrite; restore from backup instead
    }
}

When it happens

Trigger: At client.go:216-219, md5.Sum(data) is compared to the stored checksum; mismatch occurs when bytes differ — silent corruption in COS, a proxy modifying the stream, an object overwritten by a non-Terraform process that changed content but not the header, or an in-flight concurrent write that produced a torn read.

Common situations: Two Terraform runs writing the same key concurrently without locking; an external process or lifecycle rule rewrote the object; encoding/corruption from a misconfigured CDN or transparent proxy (e.g. gzip mangling); rare bit-rot or storage-tier corruption; an object uploaded by a tool that set a wrong X-Cos-Meta-Md5.

Related errors

• failed to open file at %v: checksum %s invalid
• failed to lock azure state: %s
• state blob is already locked
• blob metadata %q was empty
• failed to lock state in Consul: %s

AI-assisted analysis of hashicorp/terraform@c9def3e214 (2026-08-07). Data as JSON: /api/errors/36281c8b414772a3. Report an issue: GitHub.