{"record":{"id":"9ea3c904700c8109","repo":"hashicorp/terraform","slug":"failed-to-clean-up-file-lock-after-dynamodb-lock-e","errorCode":null,"errorMessage":"failed to clean up file lock after DynamoDB lock error: %v; original error: %w","messagePattern":"failed to clean up file lock after DynamoDB lock error: (.+?); original error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/backend/remote-state/s3/client.go","lineNumber":336,"sourceCode":"\t\tlog.Info(\"Attempting to lock remote state (DynamoDB only)...\")\n\t\tif err := c.lockWithDynamoDB(ctx, info); err != nil {\n\t\t\treturn \"\", err\n\t\t}\n\n\t\tlog.Info(\"Locked remote state (DynamoDB only)\")\n\t\treturn info.ID, nil\n\t}\n\n\t// double locking: dynamodb + file (design decision: both must succeed)\n\tlog.Info(\"Attempting to lock remote state (S3 Native and DynamoDB)...\")\n\tif err := c.lockWithFile(ctx, info, log); err != nil {\n\t\treturn \"\", err\n\t}\n\n\tif err := c.lockWithDynamoDB(ctx, info); err != nil {\n\t\t// Release the file lock if attempting to acquire the DynamoDB lock fails.\n\t\tif unlockErr := c.unlockWithFile(ctx, info.ID, &statemgr.LockError{}, log); unlockErr != nil {\n\t\t\treturn \"\", fmt.Errorf(\"failed to clean up file lock after DynamoDB lock error: %v; original error: %w\", unlockErr, err)\n\t\t}\n\n\t\treturn \"\", err\n\t}\n\n\tlog.Info(\"Locked remote state (S3 Native and DynamoDB)\")\n\treturn info.ID, nil\n}\n\n// lockWithFile attempts to acquire a lock on the remote state by uploading a lock file to Amazon S3.\n//\n// This method is used when the S3 native locking mechanism is in use. It uploads a lock file (JSON)\n// to an S3 bucket to establish a lock on the state file. If the lock file does not already\n// exist, the operation will succeed, acquiring the lock. If the lock file already exists, the operation\n// will fail due to a conditional write, indicating that the lock is already held by another Terraform client.\nfunc (c *RemoteClient) lockWithFile(ctx context.Context, info *statemgr.LockInfo, log hclog.Logger) error {\n\tlockFileJson, err := json.Marshal(info)\n\tif err != nil {","sourceCodeStart":318,"sourceCodeEnd":354,"githubUrl":"https://github.com/hashicorp/terraform/blob/d32a084675427f5ac3f7d2868578ef8b2c1dc525/internal/backend/remote-state/s3/client.go#L318-L354","documentation":"Only reachable when BOTH S3 native file locking and DynamoDB locking are enabled (useLockFile=true and ddbTable set). Terraform acquires the file lock first, then the DynamoDB lock; if the DynamoDB lock fails it attempts to roll back (release) the file lock. This error means the DynamoDB lock acquisition failed AND the compensating file-unlock also failed, so the system is left holding an S3 lock with no DynamoDB lock — the rollback itself broke.","triggerScenarios":"lockWithDynamoDB at client.go:333 fails AND the cleanup unlockWithFile at client.go:335 also fails. Triggers: DynamoDB ConditionalCheckFailed (someone else holds the DDB lock) combined with an S3 problem during cleanup (GetObject on the lock file fails, permissions revoked mid-run, or the lock file was concurrently deleted by another client).","commonSituations":"Two operators applying simultaneously where the loser hits DDB ConditionalCheckFailed and then a transient S3 blunder prevents cleanup; IAM permissions narrowed between the lock and unlock steps; SSE-C key rotated between operations so the cleanup GetObject fails decryption.","solutions":["Run `terraform force-unlock <id>` with the lock ID shown in the DynamoDB failure to clear the orphaned S3 lock file.","Inspect the S3 lock file directly: `aws s3api get-object --bucket <bucket> --key <path>.tflock -` to confirm which client owns it and whether the ID matches.","Verify the SSE-C customer key is stable across operations if customer_encryption_key is set — it must be identical for lock and unlock.","Confirm the IAM principal still has s3:GetObject + s3:DeleteObject on the lock key at unlock time (a policy change mid-run is a common cause).","If the DynamoDB lock was the original failure, address that first (it is the %w root cause); clearing DDB may be needed too."],"exampleFix":"# clear the orphaned S3 lock file left behind by the failed rollback\nterraform force-unlock <lock-id>\n\n# or manually delete the .tflock object if the ID is unknown\naws s3api delete-object --bucket tf-state-prod --key prod/terraform.tflock.tflock","handlingStrategy":"retry","validationCode":"// Validate symmetric access (lock + unlock) before enabling dual locking.\nfunc validateDualLockAccess(ctx context.Context, s3c *s3.Client, ddb *dynamodb.Client, bucket, lockKey, table string) error {\n  if _, err := s3c.HeadObject(ctx, &s3.HeadObjectInput{Bucket: &bucket, Key: &lockKey}); err != nil {\n    // NoSuchKey is fine; AccessDenied is not\n    var apiErr smithy.APIError\n    if errors.As(err, &apiErr) && apiErr.ErrorCode() != \"NotFound\" { return err }\n  }\n  if _, err := ddb.DescribeTable(ctx, &dynamodb.DescribeTableInput{TableName: &table}); err != nil {\n    return err\n  }\n  return nil\n}","typeGuard":null,"tryCatchPattern":"// On rollback failure, surface BOTH errors and emit the lock ID so force-unlock is possible.\nif unlockErr := c.unlockWithFile(ctx, info.ID, &statemgr.LockError{}, log); unlockErr != nil {\n  return \"\", fmt.Errorf(\n    \"failed to clean up file lock after DynamoDB lock error: %v; original error: %w; \" +\n    \"run `terraform force-unlock %s` to recover\", unlockErr, err, info.ID)\n}","preventionTips":["Avoid enabling both use_lockfile and dynamodb_table unless you need belt-and-suspenders locking — each adds a failure surface.","Keep IAM symmetric: the role needs Get+Put+Delete on the S3 lock key AND Put+Get+Delete on the DDB row.","Never rotate SSE-C customer keys between lock and unlock within a single apply.","Educate operators to use `terraform force-unlock <id>` rather than manually deleting locks."],"tags":["locking","s3","dynamodb","remote-state","rollback","consistency","race-condition"],"backgroundTag":null,"analyzedSha":"d32a084675427f5ac3f7d2868578ef8b2c1dc525","analyzedAt":"2026-08-11T18:43:52.779Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}