apache/hadoop · error · IllegalArgumentException

Attempted to use QJM output buffer capacity (" + size + ") g

Error message

Attempted to use QJM output buffer capacity (" + size + ") greater than the IPC max data length (ipc.maximum.data.length = " + ipcMaxDataLength + "). This will cause journals to reject edits.

What it means

QuorumJournalManager batches edits into one RPC per write. If the edit-log output buffer capacity were >= ipc.maximum.data.length (default 64MB), a full buffer would exceed the IPC frame limit and JournalNodes would reject the edits RPC. setOutputBufferCapacity() therefore rejects the configuration up front with IllegalArgumentException rather than corrupting edit flow at runtime.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:457

        writeTxnsTimeoutMs, layoutVersion);
  }

  @Override
  public void finalizeLogSegment(long firstTxId, long lastTxId)
      throws IOException {
    QuorumCall<AsyncLogger,Void> q = loggers.finalizeLogSegment(
        firstTxId, lastTxId);
    loggers.waitForWriteQuorum(q, finalizeSegmentTimeoutMs,
        String.format("finalizeLogSegment(%s-%s)", firstTxId, lastTxId));
  }

  @Override
  public void setOutputBufferCapacity(int size) {
    int ipcMaxDataLength = conf.getInt(
        CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
        CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT);
    if (size >= ipcMaxDataLength) {
      throw new IllegalArgumentException("Attempted to use QJM output buffer "
          + "capacity (" + size + ") greater than the IPC max data length ("
          + CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH + " = "
          + ipcMaxDataLength + "). This will cause journals to reject edits.");
    }
    outputBufferCapacity = size;
  }

  @Override
  public void purgeLogsOlderThan(long minTxIdToKeep) throws IOException {
    // This purges asynchronously -- there's no need to wait for a quorum
    // here, because it's always OK to fail.
    LOG.info("Purging remote journals older than txid " + minTxIdToKeep);
    loggers.purgeLogsOlderThan(minTxIdToKeep);
  }

  @Override
  public void recoverUnfinalizedSegments() throws IOException {
    Preconditions.checkState(!isActiveWriter, "already active writer");

View on GitHub (pinned to 2add963021)

Solutions

Raise ipc.maximum.data.length on the NameNode and all JournalNodes above the edit-log output buffer capacity.
Alternatively lower the edit output buffer capacity below the IPC limit.
Apply the setting consistently on NN and JNs, restart, and confirm edits flow (check for rejected IPC frames).

Example fix

<!-- before: buffer >= ipc limit -->
<property><name>ipc.maximum.data.length</name><value>8388608</value></property>

<!-- after: ipc limit comfortably above edit buffer capacity -->
<property><name>ipc.maximum.data.length</name><value>67108864</value></property>

Defensive patterns

Strategy: validation

Validate before calling

// Startup validation: reject configs where edit buffer capacity >= IPC limit
int bufferCapacity = conf.getInt("dfs.namenode.edit.log.output.buffer.capacity...", /* your edit buffer key */ 0);
int ipcMax = conf.getInt(
    CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
    CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT);
if (bufferCapacity >= ipcMax) {
  throw new IllegalArgumentException(
      "Edit buffer capacity (" + bufferCapacity + ") must be < ipc.maximum.data.length ("
          + ipcMax + "); raise ipc.maximum.data.length on NN and JournalNodes");
}

Try / catch

try {
  qjm.setOutputBufferCapacity(size);
} catch (IllegalArgumentException e) {
  // fail fast at config-load time with a clear operator message; fix config, do not catch-and-continue
  throw new ConfigurationException(
      "ipc.maximum.data.length must exceed the edit output buffer capacity; "
    + "set it above " + size + " on NameNode and JournalNodes", e);
}

Prevention

Change ipc.maximum.data.length and edit buffer settings together, cluster-wide, in config management.
Add a config lint step that asserts bufferCapacity < ipc.maximum.data.length before NN start.
After changing either value, restart JournalNodes as well as the NameNode.

When it happens

Trigger: The NameNode edit-log buffer capacity is configured at or above ipc.maximum.data.length — commonly because ipc.maximum.data.length was lowered (IPC DoS hardening) below the edit buffer size, or the edit buffer was raised without raising the IPC limit.

Common situations: Security hardening passes that shrink ipc.maximum.data.length; large-edit workloads prompting buffer tuning; drift between NameNode and JournalNode config files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f5dddc7ecff76800. Report an issue: GitHub.