{"record":{"id":"1ecb1dd8a604155b","repo":"apache/hadoop","slug":"r-getvolume-null-temp-temp","errorCode":null,"errorMessage":"r.getVolume() = null, temp={temp}","messagePattern":"r\\.getVolume\\(\\) = null, temp=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1849,"sourceCode":"        throw new ReplicaAlreadyExistsException(\n            \"temp.getGenerationStamp() != expectedGs = \" + expectedGs\n                + \", temp=\" + temp);\n      }\n\n      // TODO: check writer?\n      // set writer to the current thread\n      // temp.setWriter(Thread.currentThread());\n\n      // check length\n      final long numBytes = temp.getNumBytes();\n      if (numBytes < visible) {\n        throw new IOException(numBytes + \" = numBytes < visible = \"\n            + visible + \", temp=\" + temp);\n      }\n      // check volume\n      final FsVolumeImpl v = (FsVolumeImpl) temp.getVolume();\n      if (v == null) {\n        throw new IOException(\"r.getVolume() = null, temp=\" + temp);\n      }\n\n      final ReplicaInPipeline rbw = v.convertTemporaryToRbw(b, temp);\n\n      if(rbw.getState() != ReplicaState.RBW) {\n        throw new IOException(\"Expected replica state: \" + ReplicaState.RBW\n            + \" obtained \" + rbw.getState() + \" for converting block \"\n            + b);\n      }\n      // overwrite the RBW in the volume map\n      volumeMap.add(b.getBlockPoolId(), rbw.getReplicaInfo());\n      return rbw;\n    } finally {\n      if (dataNodeMetrics != null) {\n        long convertTemporaryToRbwMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addConvertTemporaryToRbwOp(convertTemporaryToRbwMs);\n      }\n    }","sourceCodeStart":1831,"sourceCodeEnd":1867,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1831-L1867","documentation":"Last guard before the physical conversion: the temp replica must belong to a volume. If ReplicaInfo.getVolume() returns null - a state that should be impossible for a volumeMap entry - the DataNode throws IOException('r.getVolume() = null') instead of taking an NPE deeper in the conversion.","triggerScenarios":"A volumeMap entry whose replica lost its volume reference: a volume removed or failed underneath the replica, corrupted in-memory state after crash recovery, or a custom dataset inserting volume-less replicas.","commonSituations":"Storage-directory failure or disk loss mid-operation; volumeMap inconsistency after an unclean DN restart; custom FsDataset implementations not wiring volumes into ReplicaInfo.","solutions":["Check DataNode storage-directory health and logs - a failed/unwritable volume is the usual root cause","Restart the DataNode so the volumeMap is rebuilt from on-disk state","If a custom fsdataset is in use, ensure every replica added to the volumeMap carries its volume","If volumes keep failing, retire/replace the disk before HDFS marks replicas missing (dfs.datanode.failed.volumes.tolerated)"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r == null || r.getVolume() == null) {\n  // volumeMap is unhealthy: fail over to other replicas and alert on the storage\n  failOverToOtherReplicas();\n  alertDatanodeStorageFailure();\n  return;\n}\nfsDataset.convertTemporaryToRbw(b, visible);","typeGuard":"boolean hasVolume(Replica r) {\n  return r != null && r.getVolume() != null;\n}","tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().contains(\"getVolume() = null\")) {\n    failOverToOtherReplicas();\n    alertDatanodeStorageFailure(); // in-memory replica state is inconsistent\n  } else { throw ioe; }\n}","preventionTips":["Monitor DataNode storage directories (dfs.datanode.failed.volumes.tolerated) so volume loss is visible early","Restart DataNodes after disk replacement or failure so replica metadata is rebuilt consistently","In custom dataset code, never insert a replica into the volumeMap without a live volume reference"],"tags":["hdfs","datanode","block-write","volume-failure","invariant"],"backgroundTag":"hdfs-replica-state-invariant","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}