{"record":{"id":"31cf980a87e422c0","repo":"fluent/fluentd","slug":"invalid-chunk-found-unique-id-and-key-not-exist","errorCode":null,"errorMessage":"Invalid chunk found. unique_id and key not exist: #{@path}","messagePattern":"Invalid chunk found\\. unique_id and key not exist: #(.+?)","errorType":"exception","errorClass":"FileChunkError","httpStatus":null,"severity":"critical","filePath":"lib/fluent/plugin/buffer/file_single_chunk.rb","lineNumber":203,"sourceCode":"          staged_path = \".b#{chunk_id}.\"\n          if path.index(staged_path)\n            path.sub(staged_path, \".q#{chunk_id}.\")\n          else # for unexpected cases (ex: users rename files while opened by fluentd)\n            path + \".q#{chunk_id}.chunk\"\n          end\n        end\n\n        def restore_metadata\n          if res = self.class.unique_id_and_key_from_path(@path)\n            @unique_id = res.first\n            key = decode_key(res.last)\n            if @key\n              @metadata.variables = {@key => key}\n            else\n              @metadata.tag = key\n            end\n          else\n            raise FileChunkError, \"Invalid chunk found. unique_id and key not exist: #{@path}\"\n          end\n          @size = 0\n\n          stat = File.stat(@path)\n          @created_at = stat.ctime.to_i\n          @modified_at = stat.mtime.to_i\n        end\n\n        def restore_size(chunk_format)\n          count = 0\n          File.open(@path, 'rb') { |f|\n            if chunk_format == :msgpack\n              Fluent::MessagePackFactory.msgpack_unpacker(f).each { |d| count += 1 }\n            else\n              f.each_line { |l| count += 1 }\n            end\n          }\n          @size = count","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/buffer/file_single_chunk.rb#L185-L221","documentation":"The single-file buffer plugin (buf_file_single) encodes the buffer key and unique id into each chunk filename (prefix.<escaped-key>.b<hex-id>.<suffix> for staged, .q<hex-id>. for queued). FileSingleChunk#restore_metadata calls unique_id_and_key_from_path, which matches the basename against PATH_REGEXP to extract both; when the file matches the configured path glob but not that naming pattern, it returns nil and FileChunkError 'Invalid chunk found. unique_id and key not exist: <path>' is raised, aborting startup.","triggerScenarios":"A file exists in the buffer directory that the path glob picks up but whose name does not follow the chunk pattern: outputs from out_file using the same directory/prefix, manually renamed or copied files, backup/temp files, or leftovers written under a different buffer path/key_in_path configuration.","commonSituations":"Pointing out_file and buf_file_single at the same directory; changing the buffer path or key_in_path config without cleaning the directory; operators copying 'just in case' files into the buffer dir; restore scripts that drop foreign files in place.","solutions":["Stop fluentd and move every file that is not a fluentd buffer chunk (out_file outputs, backups, renamed files) out of the buffer directory.","Check the path setting: for buf_file_single it must contain '.*.' (e.g. /var/log/fluent/buffer.*.buf) and the directory must only contain files created with that same configuration.","If you recently changed path, key_in_path, or append settings, clean or migrate the directory before restarting."],"exampleFix":"# before: out_file and buffer share a directory\n#   <match app.**>\n#     @type file\n#     path /var/log/fluent/buffer\n#     <buffer>\n#       @type file\n#       path /var/log/fluent/buffer\n#     </buffer>\n#   </match>\n# after: separate output dir from buffer dir\n#   <match app.**>\n#     @type file\n#     path /var/log/fluent/output/app\n#     <buffer>\n#       @type file\n#       path /var/log/fluent/buffer/app.*.buf\n#     </buffer>\n#   </match>","handlingStrategy":"validation","validationCode":"# preflight: list files the single-buffer glob would pick up but that are not chunks\nBUFFER_DIR = '/var/log/fluent'\nDir.glob(\"#{BUFFER_DIR}/*.buf\").each do |f|\n  parts = File.basename(f).split('.')\n  id_part = parts[-2].to_s\n  ok = id_part.length > 1 && 'bq'.include?(id_part[0]) && id_part[1..].chars.all? { |c| c =~ /[0-9a-f]/ }\n  warn \"non-chunk file in buffer dir: #{f}\" unless ok\nend","typeGuard":null,"tryCatchPattern":"begin\n  chunk = Fluent::Plugin::Buffer::FileSingleChunk.new(metadata, path, mode, key)\nrescue Fluent::Plugin::Buffer::FileChunkError => e\n  log.error \"foreign file in buffer dir, quarantining #{path}: #{e.message}\"\n  File.rename(path, \"#{path}.foreign\")\nend","preventionTips":["Give buf_file_single its own directory; never share with out_file outputs.","When changing buffer path or key_in_path, clean/migrate the directory first.","Do not rename, copy, or 'back up' files inside the buffer directory by hand.","Ensure the path contains '.*.' as documented for single-mode buffers."],"tags":["fluentd","ruby","file-buffer","path-validation","startup-failure"],"backgroundTag":"buffer-path-format-invalid","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}