{"record":{"id":"3ab2cebf6a6c87e2","repo":"apache/beam","slug":"failed-to-both-validate-v-and-delete-v-remove-v","errorCode":null,"errorMessage":"failed to both validate %v and delete: %v (remove: %v)","messagePattern":"failed to both validate (.+?) and delete: (.+?) \\(remove: (.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/artifact/materialize.go","lineNumber":436,"sourceCode":"\t_, err := os.Stat(filename)\n\tif err != nil && !os.IsNotExist(err) {\n\t\treturn errors.Wrapf(err, \"failed to stat %v\", filename)\n\t}\n\tif err == nil {\n\t\t// File already exists. Validate or delete.\n\n\t\thash, err := computeSHA256(filename)\n\t\tif err == nil && a.Sha256 == hash {\n\t\t\t// NOTE(herohde) 10/5/2017: We ignore permissions here, because\n\t\t\t// they may differ from the requested permissions due to umask\n\t\t\t// settings on unix systems (which we in turn want to respect).\n\t\t\t// We have no good way to know what to expect and thus assume\n\t\t\t// any permissions are fine.\n\t\t\treturn nil\n\t\t}\n\n\t\tif err2 := os.Remove(filename); err2 != nil {\n\t\t\treturn errors.Errorf(\"failed to both validate %v and delete: %v (remove: %v)\", filename, err, err2)\n\t\t} // else: successfully deleted bad file.\n\t} // else: file does not exist.\n\n\tif err := os.MkdirAll(filepath.Dir(filename), 0755); err != nil {\n\t\treturn err\n\t}\n\treturn retrieve(ctx, client, a, rt, filename)\n}\n\n// retrieve retrieves the given artifact and stores it as the given filename.\n// It validates that the given SHA256 matches the content and fails otherwise.\n// It expects the file to not exist, but does not clean up on failure and\n// may leave a corrupt file.\nfunc retrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, a *jobpb.ArtifactMetadata, rt string, filename string) error {\n\tstream, err := client.GetArtifact(ctx, &jobpb.LegacyGetArtifactRequest{Name: a.Name, RetrievalToken: rt})\n\tif err != nil {\n\t\treturn err\n\t}","sourceCodeStart":418,"sourceCodeEnd":454,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/artifact/materialize.go#L418-L454","documentation":"When the destination file already exists but its SHA256 does not match the metadata (or hashing failed), Retrieve attempts to delete the stale file before re-downloading. If os.Remove also fails, this error reports both the original validation/hash error and the removal error together.","triggerScenarios":"Calling Retrieve when the local file exists AND (its content hash differs from a.Sha256 or computeSHA256 errored, e.g. file unreadable) AND os.Remove(filename) fails due to permission denied, the file being held open/immutable, or the parent directory lacking write permission.","commonSituations":"Corrupted leftover file from a previous failed run in a directory owned by another user; file marked immutable (chattr +i); container volume mounted read-only; antivirus/backup software holding the file open on Windows.","solutions":["Grant write permission on the destination directory so os.Remove can succeed (chmod/chown).","Remove the immutable attribute (chattr -i) or close processes holding the file.","Manually delete the stale artifact file and re-run; Retrieve will then download fresh.","Recompute the correct SHA256 if the metadata itself is wrong (e.g. artifact re-staged with different content)."],"exampleFix":"// before\nif err2 := os.Remove(filename); err2 != nil {\n    return errors.Errorf(\"failed to both validate %v and delete: %v (remove: %v)\", filename, err, err2)\n}\n// after (user-side cleanup)\n# chmod u+w <dest-dir> && rm -f <dest-dir>/<artifact-name>\n# then re-run Retrieve/Materialize","handlingStrategy":"fallback","validationCode":"target := filepath.Join(dest, filepath.FromSlash(md.Name))\nif fi, err := os.Stat(target); err == nil && !isWritable(fi.Mode()) {\n    return fmt.Errorf(\"cannot replace unreadable artifact %q\", target)\n}","typeGuard":null,"tryCatchPattern":"if err := artifact.Retrieve(ctx, client, md, rt, dest); err != nil {\n    if strings.Contains(err.Error(), \"failed to both validate\") {\n        // clean the stale artifact out-of-band and retry once\n        os.RemoveAll(dest)\n        os.MkdirAll(dest, 0755)\n        err = artifact.Retrieve(ctx, client, md, rt, dest)\n    }\n    return err\n}","preventionTips":["Run artifact retrieval as the same user that owns the dest directory","Avoid immutable attributes and read-only mounts for artifact caches","Clean corrupt artifact caches out-of-band before retrying jobs","Watch for processes (AV/backup) holding artifact files open"],"tags":["filesystem","permission-denied","checksum","go"],"backgroundTag":"checksum-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}