apache/hadoop · error · IllegalArgumentException

Unsupported block buffer "{name}"

Error message

Unsupported block buffer "{name}"

What it means

DataBlocks.createFactory selects the upload buffering strategy by name: 'array' (heap byte[] blocks), 'disk' (spill files on local disk) or 'bytebuffer' (direct ByteBuffer blocks). The name comes from filesystem configuration keys such as fs.s3a.fast.upload.buffer. Any other string throws IllegalArgumentException('Unsupported block buffer "<name>"').

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store/DataBlocks.java:141

   * @param keyToBufferDir Key to buffer directory config for a FS.
   * @param configuration  factory configurations.
   * @param name           factory name -the option from {@link CommonConfigurationKeys}.
   * @return the factory, ready to be initialized.
   * @throws IllegalArgumentException if the name is unknown.
   */
  public static BlockFactory createFactory(String keyToBufferDir,
      Configuration configuration,
      String name) {
    LOG.debug("Creating DataFactory of type : {}", name);
    switch (name) {
    case DATA_BLOCKS_BUFFER_ARRAY:
      return new ArrayBlockFactory(keyToBufferDir, configuration);
    case DATA_BLOCKS_BUFFER_DISK:
      return new DiskBlockFactory(keyToBufferDir, configuration);
    case DATA_BLOCKS_BYTEBUFFER:
      return new ByteBufferBlockFactory(keyToBufferDir, configuration);
    default:
      throw new IllegalArgumentException("Unsupported block buffer" +
          " \"" + name + '"');
    }
  }

  /**
   * The output information for an upload.
   * It can be one of a file, an input stream or a byteArray.
   * {@link #toByteArray()} method to be used to convert the data into byte
   * array to be done in this class as well.
   * When closed, any stream is closed. Any source file is untouched.
   */
  public static final class BlockUploadData implements Closeable {
    private final File file;
    private InputStream uploadStream;
    private byte[] byteArray;
    private boolean isClosed;

    /**

View on GitHub (pinned to 2add963021)

Solutions

1. Set the key to exactly one lowercase value: disk, array or bytebuffer
2. Prefer 'disk' for large uploads to avoid OutOfMemoryError
3. Validate the configured value against the allowed set at config load time

Example fix

<!-- before -->
<property><name>fs.s3a.fast.upload.buffer</name><value>memory</value></property>

<!-- after -->
<property><name>fs.s3a.fast.upload.buffer</name><value>disk</value></property>

static boolean isSupportedBlockBuffer(String name) {
  return Arrays.asList(
      DataBlocks.DATA_BLOCKS_BUFFER_ARRAY,
      DataBlocks.DATA_BLOCKS_BUFFER_DISK,
      DataBlocks.DATA_BLOCKS_BYTEBUFFER).contains(name);
}
// use at config load: if (!isSupportedBlockBuffer(conf.get(key))) fail(...);

Catch IllegalArgumentException from DataBlocks.createFactory during filesystem initialization; the message names the bad value, so surface it with the config key it came from.

When it happens

Trigger: Setting fs.s3a.fast.upload.buffer (or the equivalent buffering key of another store) to an unknown value: 'memory', 'ByteBuffer' (wrong case), 'disk-buffer', or a value with trailing whitespace.

Common situations: Config snippets copied from older or newer Hadoop docs; XML with typos or wrong case; values taken from another store's vocabulary.

Related errors

• write (b[{b.length}], {off}, {len})
• Expecting arguments size of at most two, getting {}
• This committer does not work with the filesystem of type {sc
• E_NORMAL_FS
• No more entry in " + f

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6c4710fffa7a761b. Report an issue: GitHub.