fluent/fluentd · error · FileChunkError

invalid unique_id

Error message

invalid unique_id

What it means

After restoring a .meta payload into a Hash, restore_metadata validates required fields; :id is the chunk's unique_id. An empty Hash (the rescue fallback for unparseable data) or a Hash missing :id raises FileChunkError 'invalid unique_id'. This is the most common manifestation of a corrupt or foreign-format .meta file.

Source

Thrown at lib/fluent/plugin/buffer/file_chunk.rb:223

        end

        # used only for queued v0.12 buffer path or broken files
        def self.unique_id_from_path(path)
          if /\.(b|q)([0-9a-f]+)\.[^\/]*\Z/n =~ path # //n switch means explicit 'ASCII-8BIT' pattern
            return $2.scan(/../).map{|x| x.to_i(16) }.pack('C*')
          end
          nil
        end

        def restore_metadata(bindata)
          data = restore_metadata_with_new_format(bindata)

          unless data
            # old type of restore
            data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(bindata).read rescue {}
          end
          raise FileChunkError, "invalid meta data" if data.nil? || !data.is_a?(Hash)
          raise FileChunkError, "invalid unique_id" unless data[:id]
          raise FileChunkError, "invalid created_at" unless data[:c].to_i > 0
          raise FileChunkError, "invalid modified_at" unless data[:m].to_i > 0

          now = Fluent::Clock.real_now

          @unique_id = data[:id]
          @size = data[:s] || 0
          @created_at = data[:c]
          @modified_at = data[:m]

          @metadata.timekey = data[:timekey]
          @metadata.tag = data[:tag]
          @metadata.variables = data[:variables]
          @metadata.seq = data[:seq] || 0
        end

        def restore_metadata_partially(chunk)
          @unique_id = self.class.unique_id_from_path(chunk.path) || @unique_id

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Accept fluentd's automatic quarantine/deletion of the broken chunk on resume (data in that chunk is lost) — check logs for 'staged meta file is broken. invalid unique_id'
2. Restore the buffer directory from a known-good backup if the data is valuable
3. Address the root cause: avoid SIGKILL, monitor disk space, use a journaling filesystem
4. If it recurs across many files, suspect hardware or a version downgrade and inspect the .meta bytes (msgpack) of one file

Dir.glob('buffer/**/*.meta').each do |m|
  data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true)
                .feed(File.binread(m)).read rescue nil
  warn "#{m}: missing unique_id" unless data.is_a?(Hash) && data[:id]
end

begin
  stage, queue = buffer.resume
rescue Fluent::Plugin::Buffer::FileChunkError => e
  # per-file failures are already caught inside resume; catching here guards whole-dir failures
  log.fatal "buffer resume failed: #{e.message}"
  exit!
end

When it happens

Trigger: The msgpack fallback produced {} because feed(bindata).read raised (truncated/garbage file), so data[:id] is nil; or the Hash parsed fine but genuinely lacks the id key (e.g. a .meta written by different software or a hand-crafted file).

Common situations: Power loss truncating .meta files; partial writes from a full disk; mixing buffer directories from different fluentd major versions; restoring from an inconsistent filesystem snapshot.

Related errors

• invalid meta data
• invalid created_at
• invalid modified_at
• staged file chunk is empty
• can't create buffer file for #{path}. Stop creating buffer f

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/ed6d27122edf8e11. Report an issue: GitHub.