{"record":{"id":"9f97360a0379a1b9","repo":"apache/hadoop","slug":"output-stream-is-out-of-sync-pos-and-nextoffse","errorCode":null,"errorMessage":"output stream is out of sync, pos={} and nextOffset should be{}","messagePattern":"output stream is out of sync, pos=(.+?) and nextOffset should be(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java","lineNumber":1145,"sourceCode":"    long offset = writeCtx.getOffset();\n    int count = writeCtx.getCount();\n    WriteStableHow stableHow = writeCtx.getStableHow();\n    \n    FileHandle handle = writeCtx.getHandle();\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"do write, fileHandle {} offset: {} length: {} stableHow: {}\",\n          handle.dumpFileHandle(), offset, count, stableHow.name());\n    }\n\n    try {\n      // The write is not protected by lock. asyncState is used to make sure\n      // there is one thread doing write back at any time    \n      writeCtx.writeData(fos);\n      RpcProgramNfs3.metrics.incrBytesWritten(writeCtx.getCount());\n      \n      long flushedOffset = getFlushedOffset();\n      if (flushedOffset != (offset + count)) {\n        throw new IOException(\"output stream is out of sync, pos=\"\n            + flushedOffset + \" and nextOffset should be\"\n            + (offset + count));\n      }\n      \n\n      // Reduce memory occupation size if request was allowed dumped\n      if (writeCtx.getDataState() == WriteCtx.DataState.ALLOW_DUMP) {\n        synchronized (writeCtx) {\n          if (writeCtx.getDataState() == WriteCtx.DataState.ALLOW_DUMP) {\n            writeCtx.setDataState(WriteCtx.DataState.NO_DUMP);\n            updateNonSequentialWriteInMemory(-count);\n            if (LOG.isDebugEnabled()) {\n              LOG.debug(\"After writing {} at offset {}, \" +\n                      \"updated the memory count, new value: {}\",\n                  handle.dumpFileHandle(), offset,\n                  nonSequentialWriteInMemory.get());\n            }\n          }","sourceCodeStart":1127,"sourceCodeEnd":1163,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1127-L1163","documentation":"OpenFileCtx writes each buffered WriteCtx chunk to the local back-file with writeData(fos), then verifies that the FileOutputStream's flushed offset equals offset+count of that chunk. A mismatch means the local write stream has diverged from the NFS offset sequence — an internal data-integrity invariant — and throws IOException, which surfaces to the NFS client as NFS3ERR_IO.","triggerScenarios":"Divergence between the dump/merged write ordering and file position: heavy out-of-order or overlapping WRITE requests (typical of overwrite workloads), state corruption after gateway crash recovery from the write-dump directory, or a genuine bug in the async write-back path (the comment notes asyncState exists to serialize write-back).","commonSituations":"NFS clients (databases, rsync-like tools) issuing aggressive out-of-order WRITEs; restarting the NFS gateway while dirty writes are being recovered from nfs.dump.dir; rare races in the write-back thread under load.","solutions":["Retry the failing WRITE from the client — most clients retry on NFS3ERR_IO; a clean retry often succeeds because the ctx is reset.","Restart the NFS gateway to clear the corrupted OpenFileCtx state, and clean the write dump directory (nfs.dump.dir, default /tmp/.hdfs-nfs) if it holds stale recovered writes.","If reproducible, capture gateway logs around the exception and reduce the out-of-order write workload; report as a Hadoop JIRA with logs, since the check itself firing indicates an invariant break."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"/* no caller-side precheck can predict this internal invariant;\n   keep the guard that matters: verify gateway health before heavy workloads */\n/* e.g. verify the dump dir is fresh and the gateway was cleanly restarted */","typeGuard":null,"tryCatchPattern":"try {\n    writeCtx.writeData(fos);\n    long flushed = getFlushedOffset();\n    if (flushed != offset + count) throw new IOException(\"out of sync\");\n} catch (IOException ioe) {\n    // fail this WRITE with NFS3ERR_IO so the client retries the operation;\n    // the retry re-enters with a fresh ctx. If the error repeats, close the\n    // OpenFileCtx (the gateway discards buffered state) and let the client\n    // re-open + rewrite the range.\n}","preventionTips":["Let NFS clients retry on NFS3ERR_IO — a single occurrence usually clears on retry with reset context.","Restart the NFS gateway and clean nfs.dump.dir after crashes so recovery never resumes from stale dump state.","Keep the gateway current: this is an internal invariant; report reproducible cases with logs to the Hadoop project.","For write paths that stress out-of-order WRITEs, mount clients with options that reduce wild ordering where possible."],"tags":["nfs-gateway","hdfs","write-path","data-integrity","invariant","nfs3"],"backgroundTag":"stream-offset-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}