apache/beam · error · IllegalArgumentException

Expected file path but received directory path

Error message

Expected file path but received directory path %s

What it means

HadoopFileSystem.matchNewResource() rejects a resource spec that ends with '/' (a directory-style path) when the caller asserts it is a file via isDirectory=false. In Hadoop path conventions a trailing slash denotes a directory, so the assertion contradicts the spec.

Solutions

Strip the trailing '/' from the spec before requesting a file resource.
Pass isDirectory=true if the path genuinely refers to a directory.
Build paths with a path-join utility rather than manual string concatenation.

Example fix

// before
matchNewResource("hdfs://nn/data/output/", false);
// after
matchNewResource("hdfs://nn/data/output/part-0000.txt", false);

Defensive patterns

Strategy: validation

Validate before calling

if (singleResourceSpec.endsWith("/") && !isDirectory) {
  singleResourceSpec = singleResourceSpec.replaceAll("/+$", "");
}

Type guard

static boolean isFilePathSpec(String spec) { return !spec.endsWith("/"); }

Try / catch

try {
  resource = fileSystem.matchNewResource(spec, false);
} catch (IllegalArgumentException e) {
  resource = fileSystem.matchNewResource(spec.replaceAll("/+$", ""), false);
}

Prevention

Normalize paths: strip trailing '/' for files, add for directories
Use path-join helpers instead of string concatenation
Pass isDirectory consistently with the path's trailing slash

When it happens

Trigger: Calling HadoopFileSystem.matchNewResource(spec, false) — directly or via match/glob resolution — with singleResourceSpec ending in '/', or creating/reading a 'file' resource whose spec was built with a trailing slash.

Common situations: String-concatenating output file paths like base + "/" for filenames; path templates that already end with a slash; confusion between directory and file resource specs when resolving matches.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d009280cebb47b23. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java:327

                "Unable to create target directory %s. No further information provided by underlying filesystem.",
                targetDirectory));
      }
    }
  }

  @Override
  protected void delete(Collection<HadoopResourceId> resourceIds) throws IOException {
    for (HadoopResourceId resourceId : resourceIds) {
      // ignore response as issues are surfaced with exception
      final Path resourcePath = resourceId.toPath();
      resourcePath.getFileSystem(configuration).delete(resourceId.toPath(), false);
    }
  }

  @Override
  protected HadoopResourceId matchNewResource(String singleResourceSpec, boolean isDirectory) {
    if (singleResourceSpec.endsWith("/") && !isDirectory) {
      throw new IllegalArgumentException(
          String.format("Expected file path but received directory path %s", singleResourceSpec));
    }
    return !singleResourceSpec.endsWith("/") && isDirectory
        ? new HadoopResourceId(dropEmptyAuthority(singleResourceSpec + "/"))
        : new HadoopResourceId(dropEmptyAuthority(singleResourceSpec));
  }

  @Override
  protected String getScheme() {
    return scheme;
  }

  @Override
  protected void reportLineage(HadoopResourceId resourceId, Lineage lineage, LineageLevel level) {
    URI uri = resourceId.toPath().toUri();
    ImmutableList.Builder<String> segments = ImmutableList.builder();
    if (uri.getAuthority() != null && !uri.getAuthority().isEmpty()) {
      segments.add(uri.getAuthority());

View on GitHub (pinned to 12126d8942)