apache/hadoop · error · BlockPinningException

Got error, status=${status}, status message ${message}, ${lo

Error message

Got error, status=${status}, status message ${message}, ${logInfo}

What it means

In checkBlockOpStatus(), when the caller passed checkBlockPinningErr=true and the DataNode returned Status.ERROR_BLOCK_PINNED, a BlockPinningException is thrown. This path applies to writers that requested blocks pinned in memory (lazyPersist/RAM_DISK, via the pinned flag in the op request); the DataNode could not honor pinning, typically because not enough locked memory is available for RAM_DISK replicas.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java:118

  public static void checkBlockOpStatus(
          BlockOpResponseProto response,
          String logInfo) throws IOException {
    checkBlockOpStatus(response, logInfo, false);
  }

  public static void checkBlockOpStatus(BlockOpResponseProto response,
      String logInfo, boolean checkBlockPinningErr) throws IOException {
    if (response.getStatus() != Status.SUCCESS) {
      if (response.getStatus() == Status.ERROR_ACCESS_TOKEN) {
        throw new InvalidBlockTokenException(
          "Got access token error"
          + ", status message " + response.getMessage()
          + ", " + logInfo
        );
      } else if (checkBlockPinningErr
          && response.getStatus() == Status.ERROR_BLOCK_PINNED) {
        throw new BlockPinningException(
            "Got error"
            + ", status=" + response.getStatus().name()
            + ", status message " + response.getMessage()
            + ", " + logInfo
          );
      } else {
        throw new IOException(
          "Got error"
          + ", status=" + response.getStatus().name()
          + ", status message " + response.getMessage()
          + ", " + logInfo
        );
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. On the DataNode set dfs.datanode.max.locked.memory to a positive value within the OS limit and configure a RAM_DISK volume (e.g. [RAM_DISK]/mnt/ram_disk).
2. Raise the memlock ulimit for the DataNode process or systemd LimitMEMLOCK=infinity.
3. If pinning is optional, fall back to a normal (disk) storage policy instead of pinning.
4. Verify the node's health via DataNode JMX (lockedMemoryBudget) before routing lazy-persist writes to it.

Example fix

// hdfs-site.xml (DataNode)
<!-- before: no locked-memory budget -->

<!-- after -->
<property>
  <name>dfs.datanode.max.locked.memory</name>
  <value>1073741824</value>
</property>
<!-- and a RAM_DISK storage dir -->
<property>
  <name>dfs.datanode.data.dir</name>
  <value>[RAM_DISK]/mnt/ram_disk,[DISK]/mnt/disk1</value>
</property>

// Before lazy-persist writes, confirm the target has locked-memory budget via DataNode JMX
// (curl dn:9864/jmx -> FSDataset "LockedMemoryUsed"/capacity) or check dfs.datanode.max.locked.memory > 0 in config

try {
  out = dfs.create(path, EnumSet.of(CreateFlag.LAZY_PERSIST));
} catch (BlockPinningException e) {
  // node cannot pin in memory — fall back to normal DISK policy
  out = dfs.create(path);
}

When it happens

Trigger: Creating a file with LAZY_PERSIST storage policy or the pinned-blocks option (dfs.client.blockWriter... / create(...).setPinned/flag) while the DataNode lacks locked memory (dfs.datanode.max.locked.memory too small or 0), RAM_DISK volumes absent, or the OS cannot lock memory (ulimit memlock).

Common situations: Enabling centralized cache/lazy-persist features without sizing dfs.datanode.max.locked.memory; nodes missing RAM_DISK volume config; memlock ulimit not raised for the DataNode user; heterogeneous cluster where only some nodes support pinning.

Related errors

• Failed to copy {srcReplica} metadata to {dstMeta}
• Replica {replicaInfo} cannot be moved from storageType : {st
• LazyWriter fail to find or create lazy persist dir: ${lazyPe
• Failed to copy {srcReplica} block file to {dstFile}
• Failed to delete {lazypersistDir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f16b791a286b08d1. Report an issue: GitHub.