hashicorp/terraform · error

state data in OSS does not have the expected content. This

Error message

state data in OSS does not have the expected content.

This may be caused by unusually long delays in OSS processing a previous state
update.  Please wait for a minute or two and try again. If this problem
persists, and neither OSS nor TableStore are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
TableStore table to the following value: %x

What it means

The errBadChecksumFmt const (client.go:449) returned by RemoteClient.Get() (client.go:92) after the digest retry loop exhausts consistencyRetryTimeout (10s). It means the OSS object's MD5 still does not equal the Digest value stored in TableStore. OSS writes and OTS writes are separate, so eventual consistency or an interrupted Put can leave them divergent, and Get() detects this corruption.

Source

Thrown at internal/backend/remote-state/oss/client.go:92

		}

		// verify that this state is what we expect
		if expected, err := c.getMD5(); err != nil {
			log.Printf("[WARN] failed to fetch state md5: %s", err)
		} else if len(expected) > 0 && !bytes.Equal(expected, digest) {
			log.Printf("[WARN] state md5 mismatch: expected '%x', got '%x'", expected, digest)

			if testChecksumHook != nil {
				testChecksumHook()
			}

			if time.Now().Before(deadline) {
				time.Sleep(consistencyRetryPollInterval)
				log.Println("[INFO] retrying OSS RemoteClient.Get...")
				continue
			}

			return nil, diags.Append(fmt.Errorf(errBadChecksumFmt, digest))
		}

		break
	}
	return payload, diags
}

func (c *RemoteClient) Put(data []byte) tfdiags.Diagnostics {
	var diags tfdiags.Diagnostics
	bucket, err := c.ossClient.Bucket(c.bucketName)
	if err != nil {
		return diags.Append(fmt.Errorf("error getting bucket: %#v", err))
	}

	body := bytes.NewReader(data)

	var options []oss.Option
	if c.acl != "" {

View on GitHub (pinned to c9def3e214)

Solutions

  1. Wait 1-2 minutes for OSS eventual consistency and re-run terraform (the built-in 10s retry may be too short).
  2. If persistent, manually verify the OSS object then update the TableStore Digest row to the %x value printed in the error.
  3. Check for a crashed prior apply that left OSS/OTS out of sync and reconcile the digest.
  4. Confirm neither OSS nor TableStore is in a regional outage.

Example fix

// the error prints the expected MD5, e.g. a1b2c3...
// repair the OTS Digest row to that value:
// in TableStore console, edit row LockID=<bucket>/<stateFile>-md5,
// set column Digest = "a1b2c3..." (the hex from the message)
Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: compare the OSS object MD5 with the OTS Digest row yourself, and
// if mismatched, either wait or repair before invoking terraform Get.
func digestConsistent(ossMD5, otsDigest []byte) bool {
    return bytes.Equal(ossMD5, otsDigest)
}

Try / catch

// The backend already retries for 10s. For longer flaps, back off and retry
// at the caller; if still broken, repair the OTS Digest row to the printed %x.
for attempt := 0; attempt < 5; attempt++ {
    _, diags := client.Get()
    if !diags.HasErrors() { break }
    time.Sleep(30 * time.Second)
}

Prevention

When it happens

Trigger: Get() computes payload.MD5 from the OSS object, reads expected from getMD5() (OTS Digest row), and on mismatch retries every consistencyRetryPollInterval (2s) until the 10s deadline; if still mismatched, line 92 returns this error with the actual MD5 hex so the user can repair the OTS row.

Common situations: A previous Put() wrote OSS but failed before putMD5 completed (or vice-versa); OSS region experienced replication/read-after-write delay; manual edits to the state object; OTS Digest row edited by another tool; testChecksumHook triggered in tests.

Related errors

AI-assisted analysis of hashicorp/terraform@c9def3e214 (2026-08-07). Data as JSON: /api/errors/dd09e404a56431e4. Report an issue: GitHub.