apache/beam · error · IllegalArgumentException

Trying to read " + metadata.resourceId() + " which is a dire

Error message

Trying to read " + metadata.resourceId() + " which is a directory

What it means

FileIO's ReadableFile matching logic throws IllegalArgumentException when DirectoryTreatment.PROHIBIT is set and a matched resource is a directory. It lets users explicitly forbid directories in matched results instead of silently skipping them.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java:874

      builder.add(DisplayData.item("compression", getCompression().toString()));
      builder.add(DisplayData.item("directoryTreatment", getDirectoryTreatment().toString()));
    }

    /**
     * @return True if metadata is a directory and directory Treatment is SKIP.
     * @throws java.lang.IllegalArgumentException if metadata is a directory and directoryTreatment
     *     is Prohibited.
     * @throws java.lang.UnsupportedOperationException if metadata is a directory and
     *     directoryTreatment is not SKIP or PROHIBIT.
     */
    static boolean shouldSkipDirectory(
        MatchResult.Metadata metadata, DirectoryTreatment directoryTreatment) {
      if (metadata.resourceId().isDirectory()) {
        switch (directoryTreatment) {
          case SKIP:
            return true;
          case PROHIBIT:
            throw new IllegalArgumentException(
                "Trying to read " + metadata.resourceId() + " which is a directory");

          default:
            throw new UnsupportedOperationException(
                "Unknown DirectoryTreatment: " + directoryTreatment);
        }
      }

      return false;
    }

    /**
     * Converts metadata to readableFile. Make sure {@link
     * #shouldSkipDirectory(org.apache.beam.sdk.io.fs.MatchResult.Metadata,
     * org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment)} returns false before using.
     */
    static ReadableFile matchToReadableFile(
        MatchResult.Metadata metadata, Compression compression) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Change DirectoryTreatment.PROHIBIT to SKIP if directories are acceptable to ignore.
2. Narrow the file pattern so it matches only files (e.g. append an extension glob like *.json).
3. Remove/move the offending directory out of the matched path.

Example fix

// before
FileIO.match().to("gs://bucket/data/*").withDirectoryTreatment(DirectoryTreatment.PROHIBIT)
// after
FileIO.match().to("gs://bucket/data/*.json").withDirectoryTreatment(DirectoryTreatment.SKIP)

if (metadata.resourceId().isDirectory() && treatment == DirectoryTreatment.PROHIBIT) { throw new IllegalArgumentException("Pattern must not match directories"); }

try { FileIO.match()... } catch (IllegalArgumentException e) { /* narrow the glob or switch to SKIP */ }

When it happens

Trigger: FileIO.match()/matchAll() with .withDirectoryTreatment(DirectoryTreatment.PROHIBIT) while the match pattern (or a recursive match) resolves to a directory.

Common situations: Using a broad glob like 'gs://bucket/data/*' that includes subdirectories, combined with PROHIBIT to catch mistakes — often from bucket-layout changes adding new folders.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Extract file timestamp failed: got file timestamp == 0.
• Timing number 0b" + timingNumber.toString(2) + " has more th
• No proto encoding for PaneInfoCoder, always part of Windowed
• Runner does not support draining.
• cannot encode a null Integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2364fbe3d44d0be2. Report an issue: GitHub.