apache/hadoop · critical · IOException

output stream is out of sync, pos={} and nextOffset should b

Error message

output stream is out of sync, pos={} and nextOffset should be{}

What it means

OpenFileCtx writes each buffered WriteCtx chunk to the local back-file with writeData(fos), then verifies that the FileOutputStream's flushed offset equals offset+count of that chunk. A mismatch means the local write stream has diverged from the NFS offset sequence — an internal data-integrity invariant — and throws IOException, which surfaces to the NFS client as NFS3ERR_IO.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java:1145

    long offset = writeCtx.getOffset();
    int count = writeCtx.getCount();
    WriteStableHow stableHow = writeCtx.getStableHow();
    
    FileHandle handle = writeCtx.getHandle();
    if (LOG.isDebugEnabled()) {
      LOG.debug("do write, fileHandle {} offset: {} length: {} stableHow: {}",
          handle.dumpFileHandle(), offset, count, stableHow.name());
    }

    try {
      // The write is not protected by lock. asyncState is used to make sure
      // there is one thread doing write back at any time    
      writeCtx.writeData(fos);
      RpcProgramNfs3.metrics.incrBytesWritten(writeCtx.getCount());
      
      long flushedOffset = getFlushedOffset();
      if (flushedOffset != (offset + count)) {
        throw new IOException("output stream is out of sync, pos="
            + flushedOffset + " and nextOffset should be"
            + (offset + count));
      }
      

      // Reduce memory occupation size if request was allowed dumped
      if (writeCtx.getDataState() == WriteCtx.DataState.ALLOW_DUMP) {
        synchronized (writeCtx) {
          if (writeCtx.getDataState() == WriteCtx.DataState.ALLOW_DUMP) {
            writeCtx.setDataState(WriteCtx.DataState.NO_DUMP);
            updateNonSequentialWriteInMemory(-count);
            if (LOG.isDebugEnabled()) {
              LOG.debug("After writing {} at offset {}, " +
                      "updated the memory count, new value: {}",
                  handle.dumpFileHandle(), offset,
                  nonSequentialWriteInMemory.get());
            }
          }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the failing WRITE from the client — most clients retry on NFS3ERR_IO; a clean retry often succeeds because the ctx is reset.
2. Restart the NFS gateway to clear the corrupted OpenFileCtx state, and clean the write dump directory (nfs.dump.dir, default /tmp/.hdfs-nfs) if it holds stale recovered writes.
3. If reproducible, capture gateway logs around the exception and reduce the out-of-order write workload; report as a Hadoop JIRA with logs, since the check itself firing indicates an invariant break.

/* no caller-side precheck can predict this internal invariant;
   keep the guard that matters: verify gateway health before heavy workloads */
/* e.g. verify the dump dir is fresh and the gateway was cleanly restarted */

try {
    writeCtx.writeData(fos);
    long flushed = getFlushedOffset();
    if (flushed != offset + count) throw new IOException("out of sync");
} catch (IOException ioe) {
    // fail this WRITE with NFS3ERR_IO so the client retries the operation;
    // the retry re-enters with a fresh ctx. If the error repeats, close the
    // OpenFileCtx (the gateway discards buffered state) and let the client
    // re-open + rewrite the range.
}

When it happens

Trigger: Divergence between the dump/merged write ordering and file position: heavy out-of-order or overlapping WRITE requests (typical of overwrite workloads), state corruption after gateway crash recovery from the write-dump directory, or a genuine bug in the async write-back path (the comment notes asyncState exists to serialize write-back).

Common situations: NFS clients (databases, rsync-like tools) issuing aggressive out-of-order WRITEs; restarting the NFS gateway while dirty writes are being recovered from nfs.dump.dir; rare races in the write-back thread under load.

Related errors

• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum of type %d with bytesPerChecks
• Trying to commit inconsistent block: id = {blockId}, expecte
• Failed to copy full contents from '{srcFile}' to '{destFile}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9f97360a0379a1b9. Report an issue: GitHub.