apache/beam · error · RuntimeException

Failed to get metadata from MatchResult: %s.

Error message

Failed to get metadata from MatchResult: %s.

What it means

Inside FileIO's ReadMatches/MatchAll expansion, converting a MatchResult to its Metadata iterable failed with an IOException; it is wrapped in a RuntimeException. This happens when the underlying filesystem cannot fetch metadata after an apparently successful match (e.g. expired listing result or backend error on metadata retrieval).

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java:373

    Collection<ResourceId> resourceIdsToDelete;
    if (Sets.newHashSet(moveOptions)
        .contains(MoveOptions.StandardMoveOptions.IGNORE_MISSING_FILES)) {
      resourceIdsToDelete =
          FluentIterable.from(matchResources(Lists.newArrayList(resourceIds)))
              .filter(matchResult -> !matchResult.status().equals(Status.NOT_FOUND))
              .transformAndConcat(
                  new Function<MatchResult, Iterable<Metadata>>() {
                    @SuppressFBWarnings(
                        value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
                        justification = "https://github.com/google/guava/issues/920")
                    @Nonnull
                    @Override
                    public Iterable<Metadata> apply(@Nonnull MatchResult input) {
                      try {
                        return Lists.newArrayList(input.metadata());
                      } catch (IOException e) {
                        throw new RuntimeException(
                            String.format("Failed to get metadata from MatchResult: %s.", input),
                            e);
                      }
                    }
                  })
              .transform(
                  new Function<Metadata, ResourceId>() {
                    @SuppressFBWarnings(
                        value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
                        justification = "https://github.com/google/guava/issues/920")
                    @Nonnull
                    @Override
                    public ResourceId apply(@Nonnull Metadata input) {
                      return input.resourceId();
                    }
                  })
              .toList();
    } else {

View on GitHub (pinned to 12126d8942)

Solutions

  1. Enable runner retry: since this throws RuntimeException, the runner may retry the failed bundle — configure max retries
  2. Check storage backend health/quotas and permissions for the matched paths
  3. Reduce time between matching and reading to avoid races with file deletion
  4. Catch and rethrow as a user-visible IOException by matching with FileSystems.match directly instead of relying on the PTransform

Example fix

// before
pipeline.apply(FileIO.match().filepattern(spec)).apply(FileIO.readMatches());
// after
// add retry/failure handling at pipeline level and monitor failed bundles
pipeline.apply("Match", FileIO.match().filepattern(spec))
       .apply("ReadMatches", FileIO.readMatches());
// and set runner retry options, e.g. DataflowPipelineOptions#setMaxNumWorkers / worker retry flags
Defensive patterns

Strategy: retry

Validate before calling

// Pre-verify metadata is retrievable for matched files before the pipeline stage
for (Metadata m : FileSystems.match(spec).get(0).metadata()) {
  FileSystems.matchSingleFileSpec(m.resourceId().toString()); // throws early if unreadable
}

Try / catch

try {
  return matched.apply(FileIO.readMatches());
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to get metadata from MatchResult")) {
    // let the runner retry the bundle; or re-match after backoff
    throw e;
  }
  throw e;
}

Prevention

When it happens

Trigger: Applying FileIO.match() followed by matchAll()/ReadMatches when input.metadata() on a MatchResult throws — typically transient backend failures (GCS/S3 errors, permissions changed between match and metadata fetch) or provider implementations that fetch metadata lazily.

Common situations: Race where files are deleted between match and metadata retrieval; transient cloud storage 5xx during pipeline execution; custom FileSystem implementations whose metadata() performs network I/O that fails.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9bb29f9914efe1c7. Report an issue: GitHub.