apache/beam · error · IOException

Unable to copy resource

Error message

Unable to copy resource %s to %s. No further information provided by underlying filesystem.

What it means

HadoopFileSystem.copy() throws this IOException when FileSystem.rename/copy calls report boolean failure without an exception — a defensive branch that should not occur in practice. The Hadoop API only returns false, giving no underlying reason.

Solutions

1. Verify each source resource exists and is readable before copying.
2. Check HDFS permissions and quota on both source and destination paths.
3. Inspect Hadoop client/NameNode logs for the suppressed cause.

for (ResourceId src : srcs) {
  if (fs.match(src.toString()).status() != MatchResult.Status.OK) {
    throw new FileNotFoundException(src.toString());
  }
}

try {
  fileSystem.copy(mode, srcs, dests);
} catch (IOException e) {
  // check source existence & HDFS health, then retry with backoff
  retryCopy(srcs, dests);
}

When it happens

Trigger: Calling HadoopFileSystem.copy(OpenMode, srcResourceIds, destResourceIds) where the underlying Hadoop FileSystem copy/rename returns false for some src/dest pair (e.g. missing source, permission issue, or dest filesystem error).

Common situations: HDFS NameNode issues, missing source files after a failed earlier stage, or quota/permission problems on the destination path where Hadoop swallows the cause and just returns false.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Channel is closed
• chunk write failed
• Could not create a temporary directory for storing…
• Could not create a temporary directory for storing…
• Could not create the new requirements file

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7bb35f63b8295549. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java:220

      // to use the inefficient implementation found in FileUtil which copies all the bytes through
      // the local machine.
      //
      // HDFS FileSystem does define a concat method but could only find the DFSFileSystem
      // implementing it. The DFSFileSystem implemented concat by deleting the srcs after which
      // is not what we want. Also, all the other FileSystem implementations I saw threw
      // UnsupportedOperationException within concat.
      final boolean success =
          FileUtil.copy(
              fs,
              srcResourceIds.get(i).toPath(),
              fs,
              destResourceIds.get(i).toPath(),
              false,
              true,
              fs.getConf());
      if (!success) {
        // Defensive coding as this should not happen in practice
        throw new IOException(
            String.format(
                "Unable to copy resource %s to %s. No further information provided by underlying filesystem.",
                srcResourceIds.get(i).toPath(), destResourceIds.get(i).toPath()));
      }
    }
  }

  /**
   * Renames a {@link List} of file-like resources from one location to another.
   *
   * <p>The number of source resources must equal the number of destination resources. Destination
   * resources will be created recursively.
   *
   * @param srcResourceIds the references of the source resources
   * @param destResourceIds the references of the destination resources
   * @throws FileNotFoundException if the source resources are missing. When rename throws, the
   *     state of the resources is unknown but safe: for every (source, destination) pair of
   *     resources, the following are possible: a) source exists, b) destination exists, c) source

View on GitHub (pinned to 12126d8942)