{"record":{"id":"9d713da642fa96d0","repo":"apache/hadoop","slug":"failed-to-append-to-non-existent-file-on-client","errorCode":null,"errorMessage":"failed to append to non-existent file {} on client {}","messagePattern":"failed to append to non-existent file (.+?) on client (.+?)","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java","lineNumber":1350,"sourceCode":"      for (int i = 0; i < favoredNodes.length; i++) {\n        favoredNodeStrs[i] =\n            favoredNodes[i].getHostName() + \":\" + favoredNodes[i].getPort();\n      }\n    }\n    return favoredNodeStrs;\n  }\n\n  /**\n   * Append to an existing file if {@link CreateFlag#APPEND} is present\n   */\n  private DFSOutputStream primitiveAppend(String src, EnumSet<CreateFlag> flag,\n      Progressable progress) throws IOException {\n    if (flag.contains(CreateFlag.APPEND)) {\n      HdfsFileStatus stat = getFileInfo(src);\n      if (stat == null) { // No file to append to\n        // New file needs to be created if create option is present\n        if (!flag.contains(CreateFlag.CREATE)) {\n          throw new FileNotFoundException(\n              \"failed to append to non-existent file \" + src + \" on client \"\n                  + clientName);\n        }\n        return null;\n      }\n      return callAppend(src, flag, progress, null);\n    }\n    return null;\n  }\n\n  /**\n   * Same as {{@link #create(String, FsPermission, EnumSet, short, long,\n   *  Progressable, int, ChecksumOpt)} except that the permission\n   *  is absolute (ie has already been masked with umask.\n   */\n  public DFSOutputStream primitiveCreate(String src, FsPermission absPermission,\n      EnumSet<CreateFlag> flag, boolean createParent, short replication,\n      long blockSize, Progressable progress, int buffersize,","sourceCodeStart":1332,"sourceCodeEnd":1368,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1332-L1368","documentation":"When CreateFlag.APPEND is requested, DFSClient.primitiveAppend first stats the target with getFileInfo(); if the file is absent and CreateFlag.CREATE is not also set, append cannot proceed and throws FileNotFoundException naming the src and client. Plain append never creates the file; only APPEND+CREATE has create-or-append semantics.","triggerScenarios":"fs.append(path) (or create with EnumSet.of(CreateFlag.APPEND) alone) on a file that was deleted or never written: append-based sinks starting before the first create, reruns of jobs after cleanup deleted outputs, or a rotation suffix computing a filename that does not exist yet.","commonSituations":"Log-writer/Flume style sinks in append mode whose target file was rolled away or never created; rerun-after-cleanup workflows; date-based filename rotation landing on a new name for the first time.","solutions":["If create-or-append is intended, request both flags, e.g. DistributedFileSystem.create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND), ...) — atomically appends or creates.","Otherwise create the file explicitly on first write (fs.create(path, true)), then append on subsequent writes.","Verify the path computation (rotation index, date suffix) — most 'missing' appends target the wrong name.","Catch FileNotFoundException at the append as the create trigger instead of crashing the writer."],"exampleFix":"// before\ntry (FSDataOutputStream out = fs.append(path)) { ... }\n// FileNotFoundException: failed to append to non-existent file <path> on client <name>\n\n// after: create-or-append (for a race-free variant use\n// ((DistributedFileSystem) fs).create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND), ...))\ntry (FSDataOutputStream out = fs.exists(path)\n        ? fs.append(path)\n        : fs.create(path, true)) {\n  // ...\n}","handlingStrategy":"validation","validationCode":"// create-or-append before writing\nif (!fs.exists(path)) {\n  out = fs.create(path, true);\n} else {\n  out = fs.append(path);\n}","typeGuard":null,"tryCatchPattern":"catch (FileNotFoundException e) {\n  out = fs.create(path, true); // first write creates the file, then append\n}","preventionTips":["Design sinks as create-on-first-write, append-afterwards.","Use EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND) for atomic create-or-append.","Log rotation filenames to catch suffix/rotation bugs early."],"tags":["hdfs","append","file-not-found","createflag","writer-sink"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}