apache/druid · error · IOException

Unable to transfer bytes from file

Error message

Unable to transfer bytes from file[%s] at position[%,d]

What it means

SegmentFileBuilderV10.add copies an entire source file into the container by repeatedly calling FileChannel.transferTo. If transferTo returns zero or negative bytes at a given position, the builder cannot make progress and throws this IOException rather than looping forever. It signals the source channel yielded no bytes (typically because the source shrank or the channel is not readable at that offset).

Solutions

1. Check that source files are not being modified or deleted concurrently; snapshot inputs before merging
2. Verify the source file size with ls/stat matches the size the builder recorded (src.size()); if smaller, re-generate the source file
3. Retry the merge operation; if it reproduces, inspect filesystem health (disk errors, quotas)

Example fix

// before
final long transferred = src.transferTo(position, size - position, out);
// after (guard with re-check)
if (src.size() != size) { throw new IOE("File[%s] changed size during transfer", fileToAdd); }
long transferred = src.transferTo(position, size - position, out);

// Java: check source file size before merging
long onDisk = new File(fileToAdd).length();
if (onDisk != src.size()) { throw new IOException("source size changed: " + fileToAdd); }

try {
  builder.add(name, fileToAdd);
} catch (IOException e) {
  if (e.getMessage().startsWith("Unable to transfer bytes")) {
    // re-check source and retry once after re-verifying inputs
    retryAdd(name, fileToAdd);
  } else { throw e; }
}

When it happens

Trigger: Merging delegated files (mergeDelegatedFiles -> add) where src.transferTo(position, size - position, out) returns <= 0, e.g. the source file was truncated or modified between size() and the transfer loop.

Common situations: Concurrent modification/deletion of source segment files during merge; a source file replaced with a shorter one mid-build; filesystem errors reporting success without transferring bytes.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Cannot add files of the same name, already have
• Cannot create directory
• Cannot have a comma in the name of a file, got
• Cannot list directory
• Cannot list directory

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8994d1aaa1c171cf. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/file/SegmentFileBuilderV10.java:180

    this.outputFileName = outputFileName;
    this.baseDir = baseDir;
    this.maxContainerSize = maxContainerSize;
    this.metadataCompression = metadataCompression;
    this.externalSegmentFileBuilders = new TreeMap<>();
  }

  @Override
  public void add(String name, File fileToAdd) throws IOException
  {
    try (FileInputStream fis = new FileInputStream(fileToAdd);
         FileChannel src = fis.getChannel()) {
      final long size = src.size();
      try (SegmentFileChannel out = addWithChannel(name, size)) {
        long position = 0;
        while (position < size) {
          final long transferred = src.transferTo(position, size - position, out);
          if (transferred <= 0) {
            throw new IOE("Unable to transfer bytes from file[%s] at position[%,d]", fileToAdd, position);
          }
          position += transferred;
        }
      }
    }
  }

  @Override
  public void add(String name, ByteBuffer bufferToAdd) throws IOException
  {
    try (SegmentFileChannel out = addWithChannel(name, bufferToAdd.remaining())) {
      out.write(bufferToAdd);
    }
  }

  @Override
  public SegmentFileChannel addWithChannel(final String name, final long size) throws IOException
  {

View on GitHub (pinned to 9b90983fd2)