{"record":{"id":"a1ee95c15bf318ed","repo":"apache/hadoop","slug":"failed-to-create-basepath-source-source-all","errorCode":null,"errorMessage":"Failed to create ${basePath}[source=${source}, allow-append=${allowAppend}, ${keytab}, ${principal}] -- ${ex}","messagePattern":"Failed to create (.+?)\\[source=(.+?), allow-append=(.+?), (.+?), (.+?)\\] -- (.+?)","errorType":"exception","errorClass":"MetricsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java","lineNumber":284,"sourceCode":"\n  /**\n   * Initialize the connection to HDFS and create the base directory. Also\n   * launch the flush thread.\n   */\n  private boolean initFs() {\n    boolean success = false;\n\n    fileSystem = getFileSystem();\n\n    // This step isn't strictly necessary, but it makes debugging issues much\n    // easier. We try to create the base directory eagerly and fail with\n    // copious debug info if it fails.\n    try {\n      fileSystem.mkdirs(basePath);\n      success = true;\n    } catch (Exception ex) {\n      if (!ignoreError) {\n        throw new MetricsException(\"Failed to create \" + basePath + \"[\"\n            + SOURCE_KEY + \"=\" + source + \", \"\n            + ALLOW_APPEND_KEY + \"=\" + allowAppend + \", \"\n            + stringifySecurityProperty(KEYTAB_PROPERTY_KEY) + \", \"\n            + stringifySecurityProperty(USERNAME_PROPERTY_KEY)\n            + \"] -- \" + ex.toString(), ex);\n      }\n    }\n\n    if (success) {\n      // If we're permitted to append, check if we actually can\n      if (allowAppend) {\n        allowAppend = checkAppend(fileSystem);\n      }\n\n      flushTimer = new Timer(\"RollingFileSystemSink Flusher\", true);\n      setInitialFlushTime(new Date());\n    }\n","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java#L266-L302","documentation":"RollingFileSystemSink.initFs() deliberately creates the configured basepath directory eagerly (FileSystem.mkdirs) so misconfiguration fails fast with debug context appended to the message: source, allow-append, keytab-key and principal-key values. Any exception from mkdirs — permissions, HDFS NameNode unavailable, unwritable parent — throws MetricsException(\"Failed to create <basePath>[...]\") unless the sink's ignore-error property is true, in which case startup continues without the sink.","triggerScenarios":"basepath points to a location the daemon user cannot create directories in (e.g. /user/root/metrics); HDFS NameNode down or in safe mode at daemon startup; basepath on a filesystem with restrictive mkdir semantics (some object stores).","commonSituations":"Setting basepath to a user directory that was never pre-created; daemons racing HDFS startup; shared basepath with conflicting ownership across nodes; forgetting that the default basepath is /tmp on the DEFAULT filesystem, not on HDFS unless configured.","solutions":["Pre-create the directory with correct ownership: hdfs dfs -mkdir -p /metrics/nn && hdfs dfs -chown hdfs:hdfs /metrics/nn","Verify as the daemon user: sudo -u hdfs hdfs dfs -mkdir /metrics/nn/test","Ensure HDFS (or the target filesystem) is reachable and out of safe mode before daemons start","As a stopgap set <prefix>.sink.<instance>.ignore-error=true so the daemon runs while metrics are lost"],"exampleFix":"# before\nnamenode.sink.rolling.basepath=/user/root/metrics  # daemon user 'hdfs' cannot mkdir under /user/root\n\n# after\nnamenode.sink.rolling.basepath=/metrics/nn  # pre-created: hdfs dfs -mkdir -p /metrics/nn; hdfs dfs -chown hdfs:hdfs /metrics/nn","handlingStrategy":"validation","validationCode":"FileSystem fs = FileSystem.get(conf);\nPath base = new Path(\"/metrics/nn\");\nif (!fs.exists(base) && !fs.mkdirs(base)) {\n  throw new IllegalStateException(\"Cannot create metrics basepath \" + base\n      + \" as user \" + UserGroupInformation.getLoginUser());\n}\nif (!fs.getFileStatus(base).getPermission().getOtherAction().implies(FsAction.WRITE)\n    && !UserGroupInformation.getLoginUser().getShortUserName()\n        .equals(fs.getFileStatus(base).getOwner())) {\n  LOG.warn(\"Daemon user may lack write permission on {}\", base);\n}","typeGuard":null,"tryCatchPattern":"try {\n  sink.init(subsetConf);\n} catch (MetricsException e) {\n  // message carries basePath, source, allow-append, keytab and principal context\n  LOG.error(\"RollingFileSystemSink basepath creation failed: {}\", e.getMessage(), e.getCause());\n}","preventionTips":["Pre-create the basepath directory in cluster setup with ownership for the daemon user","Remember the default basepath is /tmp on the DEFAULT filesystem — set basepath explicitly for HDFS","Start metrics-reporting daemons only after the target filesystem is out of safe mode"],"tags":["metrics2","rolling-file-sink","hdfs","permissions","configuration"],"backgroundTag":"permission-denied","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}