apache/beam · error · IllegalStateException

Failed to validate %s

Error message

Failed to validate %s

What it means

When TFRecordIO.Read has validate=true (the default), expand() calls FileSystems.match() on the filepattern; if matching raises an IOException (filesystem unreachable, bad scheme, credentials problem), it wraps it in this IllegalStateException. It signals that the input files could not be verified to exist before the pipeline runs.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java:196

    }

    @Override
    public PCollection<byte[]> expand(PBegin input) {
      if (getFilepattern() == null) {
        throw new IllegalStateException(
            "Need to set the filepattern of a TFRecordIO.Read transform");
      }

      if (getValidate()) {
        checkState(getFilepattern().isAccessible(), "Cannot validate with a RVP.");
        try {
          MatchResult matches = FileSystems.match(getFilepattern().get());
          checkState(
              !matches.metadata().isEmpty(),
              "Unable to find any files matching %s",
              getFilepattern().get());
        } catch (IOException e) {
          throw new IllegalStateException(
              String.format("Failed to validate %s", getFilepattern().get()), e);
        }
      }

      return input.apply("Read", org.apache.beam.sdk.io.Read.from(getSource()));
    }

    // Helper to create a source specific to the requested compression type.
    protected FileBasedSource<byte[]> getSource() {
      return CompressedSource.from(new TFRecordSource(getFilepattern()))
          .withCompression(getCompression());
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {
      super.populateDisplayData(builder);
      builder
          .add(

View on GitHub (pinned to 12126d8942)

Solutions

1. Fix the filepattern so it matches an accessible location and check the wrapped IOException cause for the root reason.
2. Verify a FileSystem for the URI scheme is registered (add beam-sdks-java-io-... dependency for GCS/S3).
3. Set .withValidate(false) only if you accept deferring existence checks to runtime.

Example fix

// before
TFRecordIO.read().from("gcs://wrong-scheme/data/*.tfrecord");

// after
TFRecordIO.read().from("gs://my-bucket/data/*.tfrecord");

MatchResult mr = FileSystems.match(filePattern);
if (mr.status() == MatchResult.Status.NOT_FOUND || mr.metadata().isEmpty()) {
  throw new IllegalArgumentException("No files match: " + filePattern);
}

try { ... } catch (IllegalStateException e) { if (e.getCause() instanceof IOException) { log.error("TFRecord validation IO failure for pattern", e.getCause()); } throw e; }

When it happens

Trigger: TFRecordIO.read().from(pattern) with validate enabled and FileSystems.match throwing IOException — wrong filesystem scheme, no filesystem registered for the scheme, or network/auth failure accessing GCS/S3/local path.

Common situations: Typo in gs:// or s3:// scheme; missing Hadoop/GCS connector on classpath; expired credentials; reading a local path that doesn't resolve.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Failed to validate %s
• Pipeline update will not be possible because the following t
• One or more ErrorHandlers aren't closed, and this pipeline c
• Need to set the filepattern of a TFRecordIO.Read transform
• Mismatch of length mask when reading a record. Expected %d b

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bd8fe1da418f208. Report an issue: GitHub.