apache/beam · error

failed to both validate

Error message

failed to both validate %v and delete: %v (remove: %v)

What it means

When the destination file already exists but its SHA256 does not match the metadata (or hashing failed), Retrieve attempts to delete the stale file before re-downloading. If os.Remove also fails, this error reports both the original validation/hash error and the removal error together.

Solutions

1. Grant write permission on the destination directory so os.Remove can succeed (chmod/chown).
2. Remove the immutable attribute (chattr -i) or close processes holding the file.
3. Manually delete the stale artifact file and re-run; Retrieve will then download fresh.
4. Recompute the correct SHA256 if the metadata itself is wrong (e.g. artifact re-staged with different content).

Example fix

// before
if err2 := os.Remove(filename); err2 != nil {
    return errors.Errorf("failed to both validate %v and delete: %v (remove: %v)", filename, err, err2)
}
// after (user-side cleanup)
# chmod u+w <dest-dir> && rm -f <dest-dir>/<artifact-name>
# then re-run Retrieve/Materialize

target := filepath.Join(dest, filepath.FromSlash(md.Name))
if fi, err := os.Stat(target); err == nil && !isWritable(fi.Mode()) {
    return fmt.Errorf("cannot replace unreadable artifact %q", target)
}

if err := artifact.Retrieve(ctx, client, md, rt, dest); err != nil {
    if strings.Contains(err.Error(), "failed to both validate") {
        // clean the stale artifact out-of-band and retry once
        os.RemoveAll(dest)
        os.MkdirAll(dest, 0755)
        err = artifact.Retrieve(ctx, client, md, rt, dest)
    }
    return err
}

When it happens

Trigger: Calling Retrieve when the local file exists AND (its content hash differs from a.Sha256 or computeSHA256 errored, e.g. file unreadable) AND os.Remove(filename) fails due to permission denied, the file being held open/immutable, or the parent directory lacking write permission.

Common situations: Corrupted leftover file from a previous failed run in a directory owned by another user; file marked immutable (chattr +i); container volume mounted read-only; antivirus/backup software holding the file open on Windows.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• failed to delete: (remove: )
• bad SHA256 for : , want
• can't change to old working directory
• can't change to temp directory
• can't get current working directory

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3ab2cebf6a6c87e2. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/materialize.go:436

	_, err := os.Stat(filename)
	if err != nil && !os.IsNotExist(err) {
		return errors.Wrapf(err, "failed to stat %v", filename)
	}
	if err == nil {
		// File already exists. Validate or delete.

		hash, err := computeSHA256(filename)
		if err == nil && a.Sha256 == hash {
			// NOTE(herohde) 10/5/2017: We ignore permissions here, because
			// they may differ from the requested permissions due to umask
			// settings on unix systems (which we in turn want to respect).
			// We have no good way to know what to expect and thus assume
			// any permissions are fine.
			return nil
		}

		if err2 := os.Remove(filename); err2 != nil {
			return errors.Errorf("failed to both validate %v and delete: %v (remove: %v)", filename, err, err2)
		} // else: successfully deleted bad file.
	} // else: file does not exist.

	if err := os.MkdirAll(filepath.Dir(filename), 0755); err != nil {
		return err
	}
	return retrieve(ctx, client, a, rt, filename)
}

// retrieve retrieves the given artifact and stores it as the given filename.
// It validates that the given SHA256 matches the content and fails otherwise.
// It expects the file to not exist, but does not clean up on failure and
// may leave a corrupt file.
func retrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, a *jobpb.ArtifactMetadata, rt string, filename string) error {
	stream, err := client.GetArtifact(ctx, &jobpb.LegacyGetArtifactRequest{Name: a.Name, RetrievalToken: rt})
	if err != nil {
		return err
	}

View on GitHub (pinned to 12126d8942)