apache/beam · error · RuntimeException

Failed to read from

Error message

Failed to read from: %s

What it means

FileChecksumMatcher verifies that a pipeline's output files match an expected SHA-1 checksum. getActualChecksum calls shardedFile.readFilesWithRetries with a fixed backoff; if reading ultimately fails, it wraps the exception in a RuntimeException "Failed to read from: <file>". This indicates the expected output files could not be read (missing, not yet written, or inaccessible).

Solutions

1. Ensure the pipeline result is fully completed (result.waitUntilFinish()) before asserting with FileChecksumMatcher
2. Verify the sharded file path/URI is correct and accessible from the test process; list the files manually to confirm
3. Check filesystem credentials/permissions (e.g. GCS authentication) and network availability; inspect the wrapped cause `e` for the underlying error

Example fix

// before
PipelineResult res = pipeline.run();
assertThat(res).isInstanceOf...; // asserting before completion
assertThat(files).is(null)... new FileChecksumMatcher(expected);
// after
PipelineResult res = pipeline.run().waitUntilFinish();
assertThat("/tmp/output/prefix").create(new FileChecksumMatcher(expectedSha1));

List<String> files = shardedFile.match();
if (files == null || files.isEmpty()) {
  throw new IllegalStateException("No output files present yet at: " + shardedFile);
}

try {
  assertThat("/output/path").create(new FileChecksumMatcher(expectedSha1));
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to read from")) {
    LOG.error("Output files unreadable; cause:", e.getCause());
  }
  throw e;
}

When it happens

Trigger: Using PAssert/AssertThat on a Write-to-file pipeline where the sharded output path does not exist, is not yet flushed, has wrong permissions, or the filesystem (local/GCS/HDFS) throws during readFilesWithRetries after exhausting the backoff.

Common situations: Integration tests asserting checksums before the sink finishes writing; wrong output path or temp directory; GCS credentials/network issues; using TempFileChecksumMatcher-style paths mismatched with the actual written shards.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• assert_that must be used within a beam.Pipeline context…
• At most one of --create_test and --fix_tests may be…
• Both PubsubTestClientFactory and Clock need to be specified…
• Cannot access key as parameter outside of @OnTimer method.
• Cannot access timerId as parameter outside of @OnTimer…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ede1646ea4e26e7e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java:98

  public static FileChecksumMatcher fileContentsHaveChecksum(String checksum) {
    return new FileChecksumMatcher(checksum);
  }

  @Override
  public boolean matchesSafely(ShardedFile shardedFile) {
    return getActualChecksum(shardedFile).equals(expectedChecksum);
  }

  /**
   * Computes a checksum of the given sharded file. Not safe to call until the writing is complete.
   */
  private String getActualChecksum(ShardedFile shardedFile) {
    // Load output data
    List<String> outputs;
    try {
      outputs = shardedFile.readFilesWithRetries(Sleeper.DEFAULT, BACK_OFF_FACTORY.backoff());
    } catch (Exception e) {
      throw new RuntimeException(String.format("Failed to read from: %s", shardedFile), e);
    }

    // Verify outputs. Checksum is computed using SHA-1 algorithm
    actualChecksum = computeHash(outputs);
    LOG.debug("Generated checksum: {}", actualChecksum);

    return actualChecksum;
  }

  private static String computeHash(@Nonnull List<String> strs) {
    if (strs.isEmpty()) {
      return Hashing.sha1().hashString("", StandardCharsets.UTF_8).toString();
    }

    List<HashCode> hashCodes = new ArrayList<>();
    for (String str : strs) {
      hashCodes.add(Hashing.sha1().hashString(str, StandardCharsets.UTF_8));
    }

View on GitHub (pinned to 12126d8942)