What causes "Expected [%,d] bytes, only saw [%,d], potential corruption?" in apache/druid?

Verify the writer actually wrote the declared number of bytes before closing; compare bytesWritten to size in your code and fix the write loop

apache/druid · error · IOException

Expected [%,d] bytes, only saw [%,d], potential corruption?

Error message

Expected [%,d] bytes, only saw [%,d], potential corruption?

What it means

When addWithChannel's returned channel is closed, SegmentFileBuilderV10.close compares the bytes actually written (bytesWritten) to the declared size and throws this IOException if fewer bytes were written. It catches truncated writes at container-build time before corrupt metadata is recorded. The message is a 'potential corruption' early warning, not data loss yet.

Solutions

Verify the writer actually wrote the declared number of bytes before closing; compare bytesWritten to size in your code and fix the write loop
Check upstream sources for early EOF (decompression errors, truncated input files) and re-read from a healthy source
Ensure exceptions from intermediate write() calls are not swallowed; let them propagate instead of closing with a short write

Example fix

// before
long size = src.getSize();
try (SegmentFileChannel ch = builder.addWithChannel(name, size)) {
  while (remaining > 0 && in.read(buf) != -1) { ch.write(buf); } // may end early
}
// after
long written = 0;
try (SegmentFileChannel ch = builder.addWithChannel(name, size)) {
  int n;
  while ((n = in.read(buf)) != -1) { ch.write(buf); written += n; }
  if (written != size) { throw new IOE("short write for %s", name); }
}

Defensive patterns

Strategy: validation

Validate before calling

// Java: verify your writes before close
long written = bytesWrittenToChannel;
if (written != declaredSize) { throw new IOException("short write: " + written + " != " + declaredSize); }

Try / catch

try {
  builder.addWithChannel(name, size).close();
} catch (IOException e) {
  if (e.getMessage().contains("potential corruption")) {
    discardPartialContainer(); // delete the in-progress container file
    throw e;
  } else { throw e; }
}

Prevention

Count bytes written yourself and compare to the declared size before closing
Check upstream streams for early EOF (decompression errors, truncated inputs)
Never swallow exceptions inside write loops; let close see consistent state

When it happens

Trigger: Writing to the SegmentFileChannel returned by addWithChannel and closing after writing fewer bytes than the declared size, e.g. a loop terminated early, a buffer flush failure swallowed upstream, or passing a size larger than the source data.

Common situations: Custom segment writer code copying from a stream that ends early (network truncation, decompression error); off-by-one bugs computing expected size; an inner write silently failing and the close path detecting the shortfall.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/58206d74f0013401. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/file/SegmentFileBuilderV10.java:285

      public boolean isOpen()
      {
        return open;
      }

      @Override
      public void close() throws IOException
      {
        if (!open) {
          return;
        }
        open = false;
        writerCurrentlyInUse = false;

        if (bytesWritten != target.currOffset - startOffset) {
          throw new ISE("Perhaps there is some concurrent modification going on?");
        }
        if (bytesWritten != size) {
          throw new IOE("Expected [%,d] bytes, only saw [%,d], potential corruption?", size, bytesWritten);
        }
        internalFiles.put(
            name,
            new SegmentInternalFileMetadata(target.fileNum, startOffset, target.currOffset - startOffset)
        );
        if (owner != null) {
          columnFiles.computeIfAbsent(owner, k -> new ArrayList<>()).add(name);
        }
        mergeDelegatedFiles();
      }
    };
  }

  @Override
  public SegmentFileBuilder getExternalBuilder(String externalFile)
  {
    return externalSegmentFileBuilders.computeIfAbsent(
        externalFile,

View on GitHub (pinned to 9b90983fd2)