apache/beam · error · IOException

Error matching file spec

Error message

Error matching file spec %s: status %s

What it means

FileSystems.matchSingleFileSpec throws IOException when the underlying MatchResult status is neither OK nor NOT_FOUND, meaning the filesystem provider could not determine match results (e.g. an error occurred listing/matching).

Solutions

1. Retry the match with exponential backoff for transient backend errors
2. Check filesystem credentials and permissions for the path's scheme
3. Log matchResult.status() to identify the provider-specific failure
4. Inspect the provider's logs (GCS client, HDFS client) for the root cause

Example fix

// before
Metadata md = FileSystems.matchSingleFileSpec(spec); // throws raw IOException
// after
try {
  Metadata md = FileSystems.matchSingleFileSpec(spec);
} catch (IOException e) {
  LOG.warn("Transient match failure for %s, retrying", spec, e);
  Metadata md = retryWithBackoff(() -> FileSystems.matchSingleFileSpec(spec));
}

// Check permissions/reachability of the scheme before matching
dryRunMatch(spec); // custom pre-check with limited retries and clear error reporting

try {
  return FileSystems.matchSingleFileSpec(spec);
} catch (IOException e) {
  if (attempt < MAX_RETRIES) return retryWithBackoff(spec, attempt + 1);
  throw new IOException("Match failed after retries for " + spec, e);
}

When it happens

Trigger: Calling FileSystems.matchSingleFileSpec when the filesystem backend returns MatchResult.Status.ERROR — e.g. GCS/S3 listing failures, permission issues surfaced by the provider, or transient network errors during match.

Common situations: GCS API throttling or transient 5xx during match; IAM/service-account permission failures on bucket listing; HDFS NameNode unavailability.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Failed to get metadata from MatchResult
• File spec not found
• OffsetRetainer: failed to read offset from . Delete the…
• Cannot estimate the row count. All the sampled lines are…
• CheckpointMark must override getOffsetLimit() if…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c5e38ee4736232c9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java:198

  }

  /**
   * Returns the {@link Metadata} for a single file resource. Expects a resource specification
   * {@code spec} that matches a single result.
   *
   * @param spec a resource specification that matches exactly one result.
   * @return the {@link Metadata} for the specified resource.
   * @throws FileNotFoundException if the file resource is not found.
   * @throws IOException in the event of an error in the inner call to {@link #match}, or if the
   *     given spec does not match exactly 1 result.
   */
  public static Metadata matchSingleFileSpec(String spec) throws IOException {
    List<MatchResult> matches = FileSystems.match(Collections.singletonList(spec));
    MatchResult matchResult = Iterables.getOnlyElement(matches);
    if (matchResult.status() == Status.NOT_FOUND) {
      throw new FileNotFoundException(String.format("File spec %s not found", spec));
    } else if (matchResult.status() != Status.OK) {
      throw new IOException(
          String.format("Error matching file spec %s: status %s", spec, matchResult.status()));
    } else {
      List<Metadata> metadata = matchResult.metadata();
      if (metadata.size() != 1) {
        throw new IOException(
            String.format(
                "Expecting spec %s to match exactly one file, but matched %s: %s",
                spec, metadata.size(), metadata));
      }
      return metadata.get(0);
    }
  }

  /**
   * Returns {@link MatchResult MatchResults} for the given {@link ResourceId resourceIds}.
   *
   * @param resourceIds {@link ResourceId resourceIds} that might be derived from {@link #match},
   *     {@link ResourceId#resolve}, or {@link ResourceId#getCurrentDirectory()}.

View on GitHub (pinned to 12126d8942)