apache/hadoop · error · MetricsException

Failed to create ${basePath}[source=${source}, allow-append=

Error message

Failed to create ${basePath}[source=${source}, allow-append=${allowAppend}, ${keytab}, ${principal}] -- ${ex}

What it means

RollingFileSystemSink.initFs() deliberately creates the configured basepath directory eagerly (FileSystem.mkdirs) so misconfiguration fails fast with debug context appended to the message: source, allow-append, keytab-key and principal-key values. Any exception from mkdirs — permissions, HDFS NameNode unavailable, unwritable parent — throws MetricsException("Failed to create <basePath>[...]") unless the sink's ignore-error property is true, in which case startup continues without the sink.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java:284

  /**
   * Initialize the connection to HDFS and create the base directory. Also
   * launch the flush thread.
   */
  private boolean initFs() {
    boolean success = false;

    fileSystem = getFileSystem();

    // This step isn't strictly necessary, but it makes debugging issues much
    // easier. We try to create the base directory eagerly and fail with
    // copious debug info if it fails.
    try {
      fileSystem.mkdirs(basePath);
      success = true;
    } catch (Exception ex) {
      if (!ignoreError) {
        throw new MetricsException("Failed to create " + basePath + "["
            + SOURCE_KEY + "=" + source + ", "
            + ALLOW_APPEND_KEY + "=" + allowAppend + ", "
            + stringifySecurityProperty(KEYTAB_PROPERTY_KEY) + ", "
            + stringifySecurityProperty(USERNAME_PROPERTY_KEY)
            + "] -- " + ex.toString(), ex);
      }
    }

    if (success) {
      // If we're permitted to append, check if we actually can
      if (allowAppend) {
        allowAppend = checkAppend(fileSystem);
      }

      flushTimer = new Timer("RollingFileSystemSink Flusher", true);
      setInitialFlushTime(new Date());
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Pre-create the directory with correct ownership: hdfs dfs -mkdir -p /metrics/nn && hdfs dfs -chown hdfs:hdfs /metrics/nn
2. Verify as the daemon user: sudo -u hdfs hdfs dfs -mkdir /metrics/nn/test
3. Ensure HDFS (or the target filesystem) is reachable and out of safe mode before daemons start
4. As a stopgap set <prefix>.sink.<instance>.ignore-error=true so the daemon runs while metrics are lost

Example fix

# before
namenode.sink.rolling.basepath=/user/root/metrics  # daemon user 'hdfs' cannot mkdir under /user/root

# after
namenode.sink.rolling.basepath=/metrics/nn  # pre-created: hdfs dfs -mkdir -p /metrics/nn; hdfs dfs -chown hdfs:hdfs /metrics/nn

FileSystem fs = FileSystem.get(conf);
Path base = new Path("/metrics/nn");
if (!fs.exists(base) && !fs.mkdirs(base)) {
  throw new IllegalStateException("Cannot create metrics basepath " + base
      + " as user " + UserGroupInformation.getLoginUser());
}
if (!fs.getFileStatus(base).getPermission().getOtherAction().implies(FsAction.WRITE)
    && !UserGroupInformation.getLoginUser().getShortUserName()
        .equals(fs.getFileStatus(base).getOwner())) {
  LOG.warn("Daemon user may lack write permission on {}", base);
}

try {
  sink.init(subsetConf);
} catch (MetricsException e) {
  // message carries basePath, source, allow-append, keytab and principal context
  LOG.error("RollingFileSystemSink basepath creation failed: {}", e.getMessage(), e.getCause());
}

When it happens

Trigger: basepath points to a location the daemon user cannot create directories in (e.g. /user/root/metrics); HDFS NameNode down or in safe mode at daemon startup; basepath on a filesystem with restrictive mkdir semantics (some object stores).

Common situations: Setting basepath to a user directory that was never pre-created; daemons racing HDFS startup; shared basepath with conflicting ownership across nodes; forgetting that the default basepath is /tmp on the DEFAULT filesystem, not on HDFS unless configured.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Error connecting to file system: ${basePath} [${ex}]
• Error creating {}
• Unrecognized flush interval: ${rollInterval}. Must be a numb
• Unrecognized unit for flush interval: ${flushUnit}. Must be
• The flush interval property must be at least 1 minute. Value

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a1ee95c15bf318ed. Report an issue: GitHub.