fluent/fluentd · error · FileChunkError

invalid meta data

Error message

invalid meta data

What it means

FileChunk#restore_metadata runs during buffer resume to rebuild state from a chunk's .meta file. It first tries the length-prefixed new format, then falls back to raw MessagePack unpacking (rescuing parse failure to {}). If the result is nil or not a Hash, the .meta payload is unreadable garbage and FileChunkError 'invalid meta data' is raised. In staged-chunk loading this is re-raised as 'staged meta file is broken. invalid meta data'.

Source

Thrown at lib/fluent/plugin/buffer/file_chunk.rb:222

          end
        end

        # used only for queued v0.12 buffer path or broken files
        def self.unique_id_from_path(path)
          if /\.(b|q)([0-9a-f]+)\.[^\/]*\Z/n =~ path # //n switch means explicit 'ASCII-8BIT' pattern
            return $2.scan(/../).map{|x| x.to_i(16) }.pack('C*')
          end
          nil
        end

        def restore_metadata(bindata)
          data = restore_metadata_with_new_format(bindata)

          unless data
            # old type of restore
            data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(bindata).read rescue {}
          end
          raise FileChunkError, "invalid meta data" if data.nil? || !data.is_a?(Hash)
          raise FileChunkError, "invalid unique_id" unless data[:id]
          raise FileChunkError, "invalid created_at" unless data[:c].to_i > 0
          raise FileChunkError, "invalid modified_at" unless data[:m].to_i > 0

          now = Fluent::Clock.real_now

          @unique_id = data[:id]
          @size = data[:s] || 0
          @created_at = data[:c]
          @modified_at = data[:m]

          @metadata.timekey = data[:timekey]
          @metadata.tag = data[:tag]
          @metadata.variables = data[:variables]
          @metadata.seq = data[:seq] || 0
        end

        def restore_metadata_partially(chunk)

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Let fluentd self-heal: on resume, FileSingleBuffer/FileBuffer rescue FileChunkError per file via handle_broken_files, which logs and moves/deletes the broken chunk, then verify data loss is acceptable
2. Restore the buffer directory from backup/snapshot if the staged data matters
3. Prevent recurrence: graceful shutdowns (SIGTERM not SIGKILL), ensure disk space monitoring on the buffer volume
4. Check the logged list of remaining chunks after a broken file is found — the code explicitly warns siblings may also be corrupted

# Pre-flight: scan a file buffer dir and report unreadable .meta files before start
require 'fluent/message_pack_factory'
Dir.glob('/var/log/fluent/buffer/*.meta').each do |m|
  bin = File.binread(m)
  ok = begin
    data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(bin).read
    data.is_a?(Hash)
  rescue
    false
  end
  puts "BROKEN: #{m}" unless ok && !data[:id].nil? && data[:c].to_i > 0 && data[:m].to_i > 0
end

begin
  chunk = Fluent::Plugin::Buffer::FileChunk.new(metadata, path, :staged)
rescue Fluent::Plugin::Buffer::FileChunkError => e
  log.error "broken chunk #{path}: #{e.message}"
  # fluentd's resume already does this: handle_broken_files quarantines/deletes it
  File.rename(path, "#{path}.broken") rescue nil
end

When it happens

Trigger: A .b*.buf.meta file that is truncated (power loss mid-write), zero-length, contains non-msgpack bytes, or was written by an incompatible fluentd version/format; the msgpack_unpacker fails or returns a non-Hash scalar, and the rescue-to-{} path yields a Hash so the failure usually surfaces as the follow-on 'invalid unique_id' unless data is nil/non-Hash.

Common situations: Hard kill or power failure while chunks are staged (especially with flush_at_shutdown and no journaling); disk corruption; copying buffer directories between hosts with different fluentd versions; running out of disk while the .meta was being rewritten.

Related errors

• invalid unique_id
• invalid created_at
• invalid modified_at
• staged file chunk is empty
• can't create buffer file for #{path}. Stop creating buffer f

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/c935de63622ae83d. Report an issue: GitHub.