apache/beam · error · IOException

Unable to read file(s) after retrying %d times

Error message

Unable to read file(s) after retrying %d times

What it means

NumberedShardedFile.readFilesWithRetries retries reading a sharded file (e.g. a Dataflow job's output shards) using a bounded BackOff. If all retries fail — because shards are missing, the pattern matches nothing, or the filesystem keeps erroring — it throws an IOException stating the retry count was exhausted, with the last exception as cause.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/NumberedShardedFile.java:138

            Iterables.getOnlyElement(FileSystems.match(Collections.singletonList(filePattern)))
                .metadata();

        LOG.debug("Found {} file(s) by matching the path: {}", files.size(), filePattern);

        if (files.isEmpty() || !checkTotalNumOfFiles(files)) {
          continue;
        }

        // Read data from file paths
        return readLines(files);
      } catch (IOException e) {
        // Ignore and retry
        lastException = e;
        LOG.warn("Error in file reading. Ignore and retry.");
      }
    } while (BackOffUtils.next(sleeper, backOff));
    // Failed after max retries
    throw new IOException(
        String.format("Unable to read file(s) after retrying %d times", MAX_READ_RETRIES),
        lastException);
  }

  /**
   * Discovers all shards of this file.
   *
   * <p>Because of eventual consistency, reads may discover no files or fewer files than the shard
   * template implies. In this case, the read is considered to have failed.
   */
  public List<String> readFilesWithRetries() throws IOException, InterruptedException {
    return readFilesWithRetries(Sleeper.DEFAULT, BACK_OFF_FACTORY.backoff());
  }

  @Override
  public String toString() {
    return String.format("%s with shard template '%s'", filePattern, shardTemplate);
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Check getCause()/logs for the underlying per-retry exceptions.
2. Verify the file pattern/shard template matches the actual output naming (e.g. shard-N-of-M format).
3. Confirm the output directory exists and contains at least one shard before calling.
4. Check filesystem permissions/quotas; wait for the writing job to finish before reading.

Example fix

// before
new NumberedShardedFile("gs://bucket/out", "").readFilesWithRetries(fs);
// after
new NumberedShardedFile("gs://bucket/out", "SSS-of-NNN").readFilesWithRetries(fs);

// before reading:
MatchResult mr = fs.match(pattern);
if (mr.status() != MatchResult.Status.OK || mr.metadata().isEmpty())
  throw new IOException("no shards matched " + pattern);

null

try { files = shardedFile.readFilesWithRetries(fs); } catch (IOException e) { LOG.error("shard read failed after retries", e.getCause()); }

When it happens

Trigger: MatchResult/file pattern fails on every retry: output directory has no shards, shard template doesn't match actual files (e.g. wrong shard name template in tests), or the filesystem returns transient errors for MAX_READ_RETRIES attempts.

Common situations: Assertions on Dataflow job outputs where the template path doesn't match produced shard filenames; missing/empty output directory; GCS permission or throttling errors persisting through the backoff window.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Error matching file spec %s: status %s
• Error completing file copies with retries, sample: from %s t
• Error trying to delete %s: %s
• Unable to fetch file %s to be used locally to create a Kafka
• dynamic: wrapped StorageException via translateStorageExcept

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/30eba6861a3e1372. Report an issue: GitHub.