apache/beam · error · IOException

Unable to match files in bucket %s, prefix %s.

Error message

Unable to match files in bucket %s, prefix %s.

What it means

GcsUtilV1.listObjects wraps failures from the GCS Storage.objects.list call into this IOException when retries (ResilientOperation with SOCKET_ERRORS RetryDeterminer) are exhausted or a non-socket error occurs. It means the library could not enumerate objects under the given bucket/prefix. The underlying API exception is chained as the cause.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:477

   */
  public Objects listObjects(
      String bucket, String prefix, @Nullable String pageToken, @Nullable String delimiter)
      throws IOException {
    // List all objects that start with the prefix (including objects in sub-directories).
    Storage.Objects.List listObject = storageClient.objects().list(bucket);
    listObject.setMaxResults(MAX_LIST_ITEMS_PER_CALL);
    listObject.setPrefix(prefix);
    listObject.setDelimiter(delimiter);

    if (pageToken != null) {
      listObject.setPageToken(pageToken);
    }

    try {
      return ResilientOperation.retry(
          listObject::execute, createBackOff(), RetryDeterminer.SOCKET_ERRORS, IOException.class);
    } catch (Exception e) {
      throw new IOException(
          String.format("Unable to match files in bucket %s, prefix %s.", bucket, prefix), e);
    }
  }

  /**
   * Returns the file size from GCS or throws {@link FileNotFoundException} if the resource does not
   * exist.
   */
  @VisibleForTesting
  List<Long> fileSizes(List<GcsPath> paths) throws IOException {
    List<StorageObjectOrIOException> results = getObjects(paths);

    ImmutableList.Builder<Long> ret = ImmutableList.builder();
    for (StorageObjectOrIOException result : results) {
      ret.add(toFileSize(result));
    }
    return ret.build();
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the chained cause for the actual GCS error status
2. Verify bucket exists and the caller has storage.objects.list permission
3. Confirm bucket name and prefix values
4. Increase/adjust the BackOff if transient socket errors are the cause

Example fix

// before
List<GcsPath> files = gcsUtil.listObjects(bucket, prefix);
// after
try {
  List<GcsPath> files = gcsUtil.listObjects(bucket, prefix);
} catch (IOException e) {
  throw new IOException("Failed to list gs://" + bucket + "/" + prefix + ": " + e.getCause(), e);
}

// pre-check bucket access
boolean ok = gcsUtil.bucketExists(GcsPath.fromUri("gs://" + bucket));

boolean isListFailure(IOException e) { return e.getMessage() != null && e.getMessage().startsWith("Unable to match files"); }

try {
  files = gcsUtil.listObjects(bucket, prefix);
} catch (IOException e) {
  LOG.error("list failed for gs://{}/{}, cause={}", bucket, prefix, e.getCause(), e);
  throw e;
}

When it happens

Trigger: Calling listObjects(bucket, prefix) when GCS list calls fail repeatedly with socket errors, when the bucket does not exist or is inaccessible (403/404 not retried), or when the list request is otherwise rejected.

Common situations: Listing a bucket that was deleted or renamed, missing storage.objects.list permission, transient network failures exhausting the backoff, or an invalid prefix.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Unable to get the file object for path %s.
• Unable to read file(s) after retrying %d times
• Failed to retrieve secret bytes
• Error executing batch GCS request
• Error completing file copies with retries, sample: from %s t

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/59634c051f511d90. Report an issue: GitHub.