fluent/fluentd · critical · FileChunkError

Invalid chunk found. unique_id and key not exist: #{@path}

Error message

Invalid chunk found. unique_id and key not exist: #{@path}

What it means

The single-file buffer plugin (buf_file_single) encodes the buffer key and unique id into each chunk filename (prefix.<escaped-key>.b<hex-id>.<suffix> for staged, .q<hex-id>. for queued). FileSingleChunk#restore_metadata calls unique_id_and_key_from_path, which matches the basename against PATH_REGEXP to extract both; when the file matches the configured path glob but not that naming pattern, it returns nil and FileChunkError 'Invalid chunk found. unique_id and key not exist: <path>' is raised, aborting startup.

Source

Thrown at lib/fluent/plugin/buffer/file_single_chunk.rb:203

          staged_path = ".b#{chunk_id}."
          if path.index(staged_path)
            path.sub(staged_path, ".q#{chunk_id}.")
          else # for unexpected cases (ex: users rename files while opened by fluentd)
            path + ".q#{chunk_id}.chunk"
          end
        end

        def restore_metadata
          if res = self.class.unique_id_and_key_from_path(@path)
            @unique_id = res.first
            key = decode_key(res.last)
            if @key
              @metadata.variables = {@key => key}
            else
              @metadata.tag = key
            end
          else
            raise FileChunkError, "Invalid chunk found. unique_id and key not exist: #{@path}"
          end
          @size = 0

          stat = File.stat(@path)
          @created_at = stat.ctime.to_i
          @modified_at = stat.mtime.to_i
        end

        def restore_size(chunk_format)
          count = 0
          File.open(@path, 'rb') { |f|
            if chunk_format == :msgpack
              Fluent::MessagePackFactory.msgpack_unpacker(f).each { |d| count += 1 }
            else
              f.each_line { |l| count += 1 }
            end
          }
          @size = count

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Stop fluentd and move every file that is not a fluentd buffer chunk (out_file outputs, backups, renamed files) out of the buffer directory.
2. Check the path setting: for buf_file_single it must contain '.*.' (e.g. /var/log/fluent/buffer.*.buf) and the directory must only contain files created with that same configuration.
3. If you recently changed path, key_in_path, or append settings, clean or migrate the directory before restarting.

Example fix

# before: out_file and buffer share a directory
#   <match app.**>
#     @type file
#     path /var/log/fluent/buffer
#     <buffer>
#       @type file
#       path /var/log/fluent/buffer
#     </buffer>
#   </match>
# after: separate output dir from buffer dir
#   <match app.**>
#     @type file
#     path /var/log/fluent/output/app
#     <buffer>
#       @type file
#       path /var/log/fluent/buffer/app.*.buf
#     </buffer>
#   </match>

# preflight: list files the single-buffer glob would pick up but that are not chunks
BUFFER_DIR = '/var/log/fluent'
Dir.glob("#{BUFFER_DIR}/*.buf").each do |f|
  parts = File.basename(f).split('.')
  id_part = parts[-2].to_s
  ok = id_part.length > 1 && 'bq'.include?(id_part[0]) && id_part[1..].chars.all? { |c| c =~ /[0-9a-f]/ }
  warn "non-chunk file in buffer dir: #{f}" unless ok
end

begin
  chunk = Fluent::Plugin::Buffer::FileSingleChunk.new(metadata, path, mode, key)
rescue Fluent::Plugin::Buffer::FileChunkError => e
  log.error "foreign file in buffer dir, quarantining #{path}: #{e.message}"
  File.rename(path, "#{path}.foreign")
end

When it happens

Trigger: A file exists in the buffer directory that the path glob picks up but whose name does not follow the chunk pattern: outputs from out_file using the same directory/prefix, manually renamed or copied files, backup/temp files, or leftovers written under a different buffer path/key_in_path configuration.

Common situations: Pointing out_file and buf_file_single at the same directory; changing the buffer path or key_in_path config without cleaning the directory; operators copying 'just in case' files into the buffer dir; restore scripts that drop foreign files in place.

Related errors

• staged meta file is broken. #{e.message}
• enqueued file chunk is empty
• enqueued meta file is broken. #{e.message}
• staged file chunk is empty
• enqueued file chunk is empty

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/31cf980a87e422c0. Report an issue: GitHub.