fluent/fluentd · critical · FileChunkError

staged meta file is broken. #{e.message}

Error message

staged meta file is broken. #{e.message}

What it means

Fluentd's classic file buffer (<buffer> @type file) stores each staged chunk as a data file plus a companion .meta file holding metadata (unique_id, created_at, modified_at, time/tag/variables). When fluentd restarts and re-opens a staged chunk, restore_metadata(@meta.read) must parse that .meta content; if it raises (truncated or garbage bytes, msgpack decode failure, missing id/created_at/modified_at fields such as 'invalid meta data', or a format written by an incompatible fluentd version), the original exception is wrapped as FileChunkError 'staged meta file is broken. <original message>'. The error aborts plugin startup, so fluentd refuses to start until the broken pair is removed or fixed.

Source

Thrown at lib/fluent/plugin/buffer/file_chunk.rb:348

          if File.exist?(@meta_path)
            raise FileChunkError, "staged file chunk is empty" if File.size(@path).zero?

            @chunk = File.open(@path, 'rb+')
            @chunk.set_encoding(Encoding::ASCII_8BIT)
            @chunk.sync = true
            @chunk.seek(0, IO::SEEK_END)
            @chunk.binmode

            @meta = File.open(@meta_path, 'rb+')
            @meta.set_encoding(Encoding::ASCII_8BIT)
            @meta.sync = true
            @meta.binmode
            begin
              restore_metadata(@meta.read)
            rescue => e
              @chunk.close
              @meta.close
              raise FileChunkError, "staged meta file is broken. #{e.message}"
            end
            @meta.seek(0, IO::SEEK_SET)

            @state = :staged
            @bytesize = @chunk.size
            @commit_position = @chunk.pos
            @adding_bytes = 0
            @adding_size = 0
          else
            # classic buffer chunk - read only chunk
            @chunk = File.open(@path, 'rb')
            @chunk.set_encoding(Encoding::ASCII_8BIT)
            @chunk.binmode
            @chunk.seek(0, IO::SEEK_SET)
            @state = :queued
            @bytesize = @chunk.size

            restore_metadata_partially(@chunk)

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Stop fluentd, identify the failing pair from the log (the b<hex>.buf whose .meta raised), move both the .buf and its .meta out of the buffer directory into a quarantine folder, and restart.
2. If that buffered data must not be lost, replay the buffer directory with the fluentd version that wrote it (or restore from backup), drain the queue, then upgrade.
3. Fix the root cause so it does not recur: free disk space, stop kill -9 in favor of SIGTERM drain, and never share one buffer path between instances.
4. Verify ownership/permissions of the buffer directory for the fluentd user.

Example fix

# before: broken pair left in place, fluentd loops on startup failure
#   /var/log/fluent/buffer/b52a1fc9....buf
#   /var/log/fluent/buffer/b52a1fc9....buf.meta   (truncated)
# after: quarantine the pair and restart
sudo systemctl stop fluentd
mkdir -p /var/backups/fluent-broken
mv /var/log/fluent/buffer/b52a1fc9*.buf /var/log/fluent/buffer/b52a1fc9*.buf.meta /var/backups/fluent-broken/
sudo systemctl start fluentd

# preflight: run as the fluentd user before starting fluentd
BUFFER_DIR = '/var/log/fluent/buffer'
Dir.glob("#{BUFFER_DIR}/*.buf.meta").each do |meta|
  data = meta.sub(/\.meta\z/, '')
  warn "orphan meta: #{meta}" unless File.exist?(data)
  warn "empty/truncated meta: #{meta}" if File.size(meta) < 16
end

begin
  chunk = Fluent::Plugin::Buffer::FileChunk.new(metadata, path, :staged)
rescue Fluent::Plugin::Buffer::FileChunkError => e
  log.error "quarantining broken staged chunk #{path}: #{e.message}"
  File.rename(path, "#{path}.broken")
  File.rename("#{path}.meta", "#{path}.meta.broken") rescue nil
end

When it happens

Trigger: Fluentd restart while staged chunk files (b<hex>.buf) exist and one of the paired .meta files cannot be parsed by FileChunk#restore_metadata: file truncated by a crash/power loss mid-write (meta is written with write_metadata on every append), ENOSPC short write on a full disk, meta written by a different fluentd version whose metadata format differs, or a buffer directory shared/edited by another process.

Common situations: kill -9, OOM kill, or power loss while events are being buffered; disk-full events on the buffer partition; upgrading fluentd across versions while old staged chunks remain; copying buffer directories between hosts; pointing two fluentd instances at the same buffer path.

Related errors

• enqueued meta file is broken. #{e.message}
• enqueued file chunk is empty
• Invalid chunk found. unique_id and key not exist: #{@path}
• staged file chunk is empty
• enqueued file chunk is empty

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/0ae68566a38d494a. Report an issue: GitHub.