fluent/fluentd · critical · FileChunkError

enqueued meta file is broken. #{e.message}

Error message

enqueued meta file is broken. #{e.message}

What it means

When loading an existing enqueued chunk, the classic file buffer reads the companion .meta (only if it exists and is readable) and runs restore_metadata on its bytes; any parse failure (truncated content, msgpack error, 'invalid meta data'/'invalid unique_id'/'invalid created_at' from the field checks) is re-raised as FileChunkError 'enqueued meta file is broken. <original message>'. Note the asymmetry: a missing or unreadable .meta is fine (restore_metadata_partially recovers the chunk from the data file alone); only a present-but-unparseable .meta raises.

Source

Thrown at lib/fluent/plugin/buffer/file_chunk.rb:390

        def load_existing_enqueued_chunk(path)
          @path = path
          raise FileChunkError, "enqueued file chunk is empty" if File.size(@path).zero?

          @chunk = File.open(@path, 'rb')
          @chunk.set_encoding(Encoding::ASCII_8BIT)
          @chunk.binmode
          @chunk.seek(0, IO::SEEK_SET)
          @bytesize = @chunk.size
          @commit_position = @chunk.size

          @meta_path = @path + '.meta'
          if File.readable?(@meta_path)
            begin
              restore_metadata(File.open(@meta_path){|f| f.set_encoding(Encoding::ASCII_8BIT); f.binmode; f.read })
            rescue => e
              @chunk.close
              raise FileChunkError, "enqueued meta file is broken. #{e.message}"
            end
          else
            restore_metadata_partially(@chunk)
          end
          @state = :queued
        end

        private

        def restore_metadata_with_new_format(chunk)
          if chunk.size <= 6 # size of BUFFER_HEADER (2) + size of data size(4)
            return nil
          end

          if chunk.slice(0, 2) == BUFFER_HEADER
            size = chunk.slice(2, 4).unpack1('N')
            if size
              return Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(chunk.slice(6, size)).read rescue nil

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Rename or delete only the broken .meta file: without it the queued chunk is still delivered via restore_metadata_partially (time/tag metadata is lost but the events ship), then restart fluentd.
2. If the data file itself is also suspect (empty/garbage), quarantine both files instead.
3. Fix the durability root cause: ensure the buffer partition has space, use graceful shutdown, and avoid version jumps with non-empty buffers.

Example fix

# before: 'enqueued meta file is broken. invalid meta data' blocks startup
# after: drop only the meta, keep the data chunk deliverable
sudo systemctl stop fluentd
mv /var/log/fluent/buffer/q3ab9....buf.meta /var/backups/fluent-broken/
sudo systemctl start fluentd

# preflight: queued chunks without meta are OK; broken meta files are the risk
BUFFER_DIR = '/var/log/fluent/buffer'
Dir.glob("#{BUFFER_DIR}/q*.buf.meta").each do |meta|
  warn "suspicious meta (too small): #{meta}" if File.size(meta) < 16
end

begin
  chunk = Fluent::Plugin::Buffer::FileChunk.new(metadata, path, :queued)
rescue Fluent::Plugin::Buffer::FileChunkError => e
  # dropping just the meta lets restore_metadata_partially ship the data
  log.warn "dropping broken meta for #{path}: #{e.message}"
  File.rename("#{path}.meta", "#{path}.meta.broken") rescue nil
  retry
end

When it happens

Trigger: A readable q<hex>.buf.meta that fails restore_metadata: truncated by crash mid-write, corrupted bytes, or written by an incompatible fluentd version, next to an otherwise valid q<hex>.buf data file.

Common situations: Crash/power loss right after the chunk data was flushed but while meta was being rewritten; version upgrades with leftover queued chunks; disk-full conditions; manual edits to buffer files.

Related errors

• staged meta file is broken. #{e.message}
• enqueued file chunk is empty
• Invalid chunk found. unique_id and key not exist: #{@path}
• staged file chunk is empty
• enqueued file chunk is empty

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/53d1f46ffd643b96. Report an issue: GitHub.