apache/hadoop · error · IOException
Join on writer thread ${thread} timed out
Error message
Join on writer thread ${thread} timed out What it means
Error "Join on writer thread ${thread} timed out" thrown in apache/hadoop.
Source
Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java:276
while (true) {
Thread thread = writer.get();
if ((thread == null) || (thread == Thread.currentThread()) ||
(!thread.isAlive())) {
if (writer.compareAndSet(thread, null)) {
return; // Done
}
// The writer changed. Go back to the start of the loop and attempt to
// stop the new writer.
continue;
}
thread.interrupt();
try {
thread.join(xceiverStopTimeout);
if (thread.isAlive()) {
// Our thread join timed out.
final String msg = "Join on writer thread " + thread + " timed out";
DataNode.LOG.warn(msg + "\n" + StringUtils.getStackTrace(thread));
throw new IOException(msg);
}
} catch (InterruptedException e) {
throw new IOException("Waiting for writer thread is interrupted.");
}
}
}
@Override // Object
public int hashCode() {
return super.hashCode();
}
@Override // ReplicaInPipeline
public ReplicaOutputStreams createStreams(boolean isCreate,
DataChecksum requestedChecksum) throws IOException {
final File blockFile = getBlockFile();
final File metaFile = getMetaFile();
if (DataNode.LOG.isDebugEnabled()) {View on GitHub (pinned to 2add963021)
Solutions
- Investigate why the writer thread did not finish (slow disk, downstream stall); check disk metrics and downstream DataNode logs.
- Increase timeouts only after ruling out hardware and network stalls.
When it happens
Trigger: Joining the replica writer thread times out after a write or finalize, because the writer is stalled.
Common situations: The replica writer thread did not stop within the join timeout, typically due to a blocked disk I/O. Check for slow or failing disks.
Understand the failure class
- Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.
AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22).
Data as JSON: /api/errors/560c5d745391694d.
Report an issue: GitHub.