{"record":{"id":"c935de63622ae83d","repo":"fluent/fluentd","slug":"invalid-meta-data","errorCode":null,"errorMessage":"invalid meta data","messagePattern":"invalid meta data","errorType":"exception","errorClass":"FileChunkError","httpStatus":null,"severity":"error","filePath":"lib/fluent/plugin/buffer/file_chunk.rb","lineNumber":222,"sourceCode":"          end\n        end\n\n        # used only for queued v0.12 buffer path or broken files\n        def self.unique_id_from_path(path)\n          if /\\.(b|q)([0-9a-f]+)\\.[^\\/]*\\Z/n =~ path # //n switch means explicit 'ASCII-8BIT' pattern\n            return $2.scan(/../).map{|x| x.to_i(16) }.pack('C*')\n          end\n          nil\n        end\n\n        def restore_metadata(bindata)\n          data = restore_metadata_with_new_format(bindata)\n\n          unless data\n            # old type of restore\n            data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(bindata).read rescue {}\n          end\n          raise FileChunkError, \"invalid meta data\" if data.nil? || !data.is_a?(Hash)\n          raise FileChunkError, \"invalid unique_id\" unless data[:id]\n          raise FileChunkError, \"invalid created_at\" unless data[:c].to_i > 0\n          raise FileChunkError, \"invalid modified_at\" unless data[:m].to_i > 0\n\n          now = Fluent::Clock.real_now\n\n          @unique_id = data[:id]\n          @size = data[:s] || 0\n          @created_at = data[:c]\n          @modified_at = data[:m]\n\n          @metadata.timekey = data[:timekey]\n          @metadata.tag = data[:tag]\n          @metadata.variables = data[:variables]\n          @metadata.seq = data[:seq] || 0\n        end\n\n        def restore_metadata_partially(chunk)","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/buffer/file_chunk.rb#L204-L240","documentation":"FileChunk#restore_metadata runs during buffer resume to rebuild state from a chunk's .meta file. It first tries the length-prefixed new format, then falls back to raw MessagePack unpacking (rescuing parse failure to {}). If the result is nil or not a Hash, the .meta payload is unreadable garbage and FileChunkError 'invalid meta data' is raised. In staged-chunk loading this is re-raised as 'staged meta file is broken. invalid meta data'.","triggerScenarios":"A .b*.buf.meta file that is truncated (power loss mid-write), zero-length, contains non-msgpack bytes, or was written by an incompatible fluentd version/format; the msgpack_unpacker fails or returns a non-Hash scalar, and the rescue-to-{} path yields a Hash so the failure usually surfaces as the follow-on 'invalid unique_id' unless data is nil/non-Hash.","commonSituations":"Hard kill or power failure while chunks are staged (especially with flush_at_shutdown and no journaling); disk corruption; copying buffer directories between hosts with different fluentd versions; running out of disk while the .meta was being rewritten.","solutions":["Let fluentd self-heal: on resume, FileSingleBuffer/FileBuffer rescue FileChunkError per file via handle_broken_files, which logs and moves/deletes the broken chunk, then verify data loss is acceptable","Restore the buffer directory from backup/snapshot if the staged data matters","Prevent recurrence: graceful shutdowns (SIGTERM not SIGKILL), ensure disk space monitoring on the buffer volume","Check the logged list of remaining chunks after a broken file is found — the code explicitly warns siblings may also be corrupted"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# Pre-flight: scan a file buffer dir and report unreadable .meta files before start\nrequire 'fluent/message_pack_factory'\nDir.glob('/var/log/fluent/buffer/*.meta').each do |m|\n  bin = File.binread(m)\n  ok = begin\n    data = Fluent::MessagePackFactory.msgpack_unpacker(symbolize_keys: true).feed(bin).read\n    data.is_a?(Hash)\n  rescue\n    false\n  end\n  puts \"BROKEN: #{m}\" unless ok && !data[:id].nil? && data[:c].to_i > 0 && data[:m].to_i > 0\nend","typeGuard":null,"tryCatchPattern":"begin\n  chunk = Fluent::Plugin::Buffer::FileChunk.new(metadata, path, :staged)\nrescue Fluent::Plugin::Buffer::FileChunkError => e\n  log.error \"broken chunk #{path}: #{e.message}\"\n  # fluentd's resume already does this: handle_broken_files quarantines/deletes it\n  File.rename(path, \"#{path}.broken\") rescue nil\nend","preventionTips":["Stop fluentd with SIGTERM, never SIGKILL, so staged chunks close cleanly","Monitor free space on the buffer volume","Do not reuse buffer dirs across fluentd major versions","Snapshot/backup the buffer directory before upgrades"],"tags":["fluentd","buffer","file-chunk","corruption","resume","msgpack"],"backgroundTag":"corrupt-buffer-metadata","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}