What causes "Invalid stream type: \"" + s + "\"" in apache/hadoop?

Set fs.s3a.input.stream.type to one of: Classic, Prefetch, Analytics, Custom, or Default (or remove the property entirely).

apache/hadoop · error · IllegalArgumentException

Invalid stream type: \"" + s + "\"

Error message

Invalid stream type: \"" + s + "\"

What it means

StreamIntegration.determineInputStreamType() parses fs.s3a.input.stream.type. Valid values are the InputStreamType enum names Classic, Prefetch, Analytics, Custom, plus 'Default'/empty (which resolve to the default stream). Anything else throws IllegalArgumentException("Invalid stream type: \"<s>\"") when the S3A filesystem initializes input streams.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/streams/StreamIntegration.java:158

    if (conf.getBoolean(PREFETCH_ENABLED_KEY, false)) {
      // prefetch enabled, warn (once) then change it to be the default.
      WARN_PREFETCH_KEY.info("Using {} is deprecated: choose the appropriate stream in {}",
          PREFETCH_ENABLED_KEY, INPUT_STREAM_TYPE);
      return InputStreamType.Prefetch;
    }

    // retrieve the enum value, returning the configured value or
    // the (calculated) default
    return ConfigurationHelper.resolveEnum(conf,
        INPUT_STREAM_TYPE,
        InputStreamType.class,
        s -> {
          if (isEmpty(s) || DEFAULT.equalsIgnoreCase(s)) {
            // return default type.
            return DEFAULT_STREAM_TYPE;
          } else {
            // any other value
            throw new IllegalArgumentException(E_INVALID_STREAM_TYPE
                + " \"" + s + "\"");
          }
        });
  }

  /**
   * Load the input stream factory defined in the option
   * {@link Constants#INPUT_STREAM_CUSTOM_FACTORY}.
   * @param conf configuration to use
   * @return the custom factory
   * @throws RuntimeException any binding/loading/instantiation problem
   */
  static ObjectInputStreamFactory loadCustomFactory(Configuration conf) {

    // make sure the classname option is actually set
    final String name = conf.getTrimmed(INPUT_STREAM_CUSTOM_FACTORY, "");
    checkArgument(!isEmpty(name), E_EMPTY_CUSTOM_CLASSNAME);

View on GitHub (pinned to 2add963021)

Solutions

Set fs.s3a.input.stream.type to one of: Classic, Prefetch, Analytics, Custom, or Default (or remove the property entirely).
If you had 'Guarded', remove it - the Guarded stream was deleted in Hadoop 3.4.0; 'Classic' is the closest replacement behavior.
Check the exact effective value: hadoop conf | grep input.stream.type, or conf.getPropertySources("fs.s3a.input.stream.type") to find which file supplies it.
For 'Custom', also set fs.s3a.input.stream.custom.factory to your factory class.

Example fix

<!-- before -->
<property><name>fs.s3a.input.stream.type</name><value>Guarded</value></property>

<!-- after (Hadoop 3.4+) -->
<property><name>fs.s3a.input.stream.type</name><value>Classic</value></property>
<!-- or simply delete the property to use the default -->

Defensive patterns

Strategy: validation

Validate before calling

Set<String> valid = Set.of("Classic", "Prefetch", "Analytics", "Custom", "Default");
String v = conf.getTrimmed("fs.s3a.input.stream.type", "");
if (!v.isEmpty() && !valid.contains(v)) {
  throw new IllegalArgumentException(
      "fs.s3a.input.stream.type='" + v + "' invalid; allowed: " + valid);
}

Try / catch

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Invalid stream type")) {
    // fix fs.s3a.input.stream.type and rebuild config before retrying
  } else { throw e; }
}

Prevention

Validate the option in a config-lint step before deploying jobs.
After a Hadoop upgrade, grep configs for removed values like 'Guarded'.
Check conf.getPropertySources(key) to find which layer supplies a bad value.

When it happens

Trigger: Setting fs.s3a.input.stream.type to a typo or removed value, e.g. 'Guarded' (removed in Hadoop 3.4.0, HADOOP-16831), 'classic ' with stray characters, 'VectoredGeneric' or 'default1'; XML config with whitespace/HTML entity issues; setting the old fs.s3a.prefetch.enabled alongside an invalid type value.

Common situations: Upgrading to Hadoop 3.4+ with configs written for 3.3.x that say fs.s3a.input.stream.type=Guarded; copy-paste from outdated documentation or blog posts; environment-level defaults (core-site.xml in the image) applied to every job; case sensitivity ('classic' vs 'Classic') depending on resolution path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8df5c7365826ff26. Report an issue: GitHub.