{"record":{"id":"ede1646ea4e26e7e","repo":"apache/beam","slug":"failed-to-read-from-s","errorCode":null,"errorMessage":"Failed to read from: %s","messagePattern":"Failed to read from: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java","lineNumber":98,"sourceCode":"  public static FileChecksumMatcher fileContentsHaveChecksum(String checksum) {\n    return new FileChecksumMatcher(checksum);\n  }\n\n  @Override\n  public boolean matchesSafely(ShardedFile shardedFile) {\n    return getActualChecksum(shardedFile).equals(expectedChecksum);\n  }\n\n  /**\n   * Computes a checksum of the given sharded file. Not safe to call until the writing is complete.\n   */\n  private String getActualChecksum(ShardedFile shardedFile) {\n    // Load output data\n    List<String> outputs;\n    try {\n      outputs = shardedFile.readFilesWithRetries(Sleeper.DEFAULT, BACK_OFF_FACTORY.backoff());\n    } catch (Exception e) {\n      throw new RuntimeException(String.format(\"Failed to read from: %s\", shardedFile), e);\n    }\n\n    // Verify outputs. Checksum is computed using SHA-1 algorithm\n    actualChecksum = computeHash(outputs);\n    LOG.debug(\"Generated checksum: {}\", actualChecksum);\n\n    return actualChecksum;\n  }\n\n  private static String computeHash(@Nonnull List<String> strs) {\n    if (strs.isEmpty()) {\n      return Hashing.sha1().hashString(\"\", StandardCharsets.UTF_8).toString();\n    }\n\n    List<HashCode> hashCodes = new ArrayList<>();\n    for (String str : strs) {\n      hashCodes.add(Hashing.sha1().hashString(str, StandardCharsets.UTF_8));\n    }","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java#L80-L116","documentation":"FileChecksumMatcher verifies that a pipeline's output files match an expected SHA-1 checksum. getActualChecksum calls shardedFile.readFilesWithRetries with a fixed backoff; if reading ultimately fails, it wraps the exception in a RuntimeException \"Failed to read from: <file>\". This indicates the expected output files could not be read (missing, not yet written, or inaccessible).","triggerScenarios":"Using PAssert/AssertThat on a Write-to-file pipeline where the sharded output path does not exist, is not yet flushed, has wrong permissions, or the filesystem (local/GCS/HDFS) throws during readFilesWithRetries after exhausting the backoff.","commonSituations":"Integration tests asserting checksums before the sink finishes writing; wrong output path or temp directory; GCS credentials/network issues; using TempFileChecksumMatcher-style paths mismatched with the actual written shards.","solutions":["Ensure the pipeline result is fully completed (result.waitUntilFinish()) before asserting with FileChecksumMatcher","Verify the sharded file path/URI is correct and accessible from the test process; list the files manually to confirm","Check filesystem credentials/permissions (e.g. GCS authentication) and network availability; inspect the wrapped cause `e` for the underlying error"],"exampleFix":"// before\nPipelineResult res = pipeline.run();\nassertThat(res).isInstanceOf...; // asserting before completion\nassertThat(files).is(null)... new FileChecksumMatcher(expected);\n// after\nPipelineResult res = pipeline.run().waitUntilFinish();\nassertThat(\"/tmp/output/prefix\").create(new FileChecksumMatcher(expectedSha1));","handlingStrategy":"retry","validationCode":"List<String> files = shardedFile.match();\nif (files == null || files.isEmpty()) {\n  throw new IllegalStateException(\"No output files present yet at: \" + shardedFile);\n}","typeGuard":null,"tryCatchPattern":"try {\n  assertThat(\"/output/path\").create(new FileChecksumMatcher(expectedSha1));\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to read from\")) {\n    LOG.error(\"Output files unreadable; cause:\", e.getCause());\n  }\n  throw e;\n}","preventionTips":["Call result.waitUntilFinish() before checksum assertions so all shards are committed","Verify the output prefix/path matches the Write transform's configured destination","Confirm filesystem credentials and permissions (local dir, GCS bucket, HDFS) before running the test"],"tags":["apache-beam","file-io","testing","checksum"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}