{"record":{"id":"0ae68566a38d494a","repo":"fluent/fluentd","slug":"staged-meta-file-is-broken-e-message","errorCode":null,"errorMessage":"staged meta file is broken. #{e.message}","messagePattern":"staged meta file is broken\\. #(.+?)","errorType":"exception","errorClass":"FileChunkError","httpStatus":null,"severity":"critical","filePath":"lib/fluent/plugin/buffer/file_chunk.rb","lineNumber":348,"sourceCode":"          if File.exist?(@meta_path)\n            raise FileChunkError, \"staged file chunk is empty\" if File.size(@path).zero?\n\n            @chunk = File.open(@path, 'rb+')\n            @chunk.set_encoding(Encoding::ASCII_8BIT)\n            @chunk.sync = true\n            @chunk.seek(0, IO::SEEK_END)\n            @chunk.binmode\n\n            @meta = File.open(@meta_path, 'rb+')\n            @meta.set_encoding(Encoding::ASCII_8BIT)\n            @meta.sync = true\n            @meta.binmode\n            begin\n              restore_metadata(@meta.read)\n            rescue => e\n              @chunk.close\n              @meta.close\n              raise FileChunkError, \"staged meta file is broken. #{e.message}\"\n            end\n            @meta.seek(0, IO::SEEK_SET)\n\n            @state = :staged\n            @bytesize = @chunk.size\n            @commit_position = @chunk.pos\n            @adding_bytes = 0\n            @adding_size = 0\n          else\n            # classic buffer chunk - read only chunk\n            @chunk = File.open(@path, 'rb')\n            @chunk.set_encoding(Encoding::ASCII_8BIT)\n            @chunk.binmode\n            @chunk.seek(0, IO::SEEK_SET)\n            @state = :queued\n            @bytesize = @chunk.size\n\n            restore_metadata_partially(@chunk)","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/buffer/file_chunk.rb#L330-L366","documentation":"Fluentd's classic file buffer (<buffer> @type file) stores each staged chunk as a data file plus a companion .meta file holding metadata (unique_id, created_at, modified_at, time/tag/variables). When fluentd restarts and re-opens a staged chunk, restore_metadata(@meta.read) must parse that .meta content; if it raises (truncated or garbage bytes, msgpack decode failure, missing id/created_at/modified_at fields such as 'invalid meta data', or a format written by an incompatible fluentd version), the original exception is wrapped as FileChunkError 'staged meta file is broken. <original message>'. The error aborts plugin startup, so fluentd refuses to start until the broken pair is removed or fixed.","triggerScenarios":"Fluentd restart while staged chunk files (b<hex>.buf) exist and one of the paired .meta files cannot be parsed by FileChunk#restore_metadata: file truncated by a crash/power loss mid-write (meta is written with write_metadata on every append), ENOSPC short write on a full disk, meta written by a different fluentd version whose metadata format differs, or a buffer directory shared/edited by another process.","commonSituations":"kill -9, OOM kill, or power loss while events are being buffered; disk-full events on the buffer partition; upgrading fluentd across versions while old staged chunks remain; copying buffer directories between hosts; pointing two fluentd instances at the same buffer path.","solutions":["Stop fluentd, identify the failing pair from the log (the b<hex>.buf whose .meta raised), move both the .buf and its .meta out of the buffer directory into a quarantine folder, and restart.","If that buffered data must not be lost, replay the buffer directory with the fluentd version that wrote it (or restore from backup), drain the queue, then upgrade.","Fix the root cause so it does not recur: free disk space, stop kill -9 in favor of SIGTERM drain, and never share one buffer path between instances.","Verify ownership/permissions of the buffer directory for the fluentd user."],"exampleFix":"# before: broken pair left in place, fluentd loops on startup failure\n#   /var/log/fluent/buffer/b52a1fc9....buf\n#   /var/log/fluent/buffer/b52a1fc9....buf.meta   (truncated)\n# after: quarantine the pair and restart\nsudo systemctl stop fluentd\nmkdir -p /var/backups/fluent-broken\nmv /var/log/fluent/buffer/b52a1fc9*.buf /var/log/fluent/buffer/b52a1fc9*.buf.meta /var/backups/fluent-broken/\nsudo systemctl start fluentd","handlingStrategy":"validation","validationCode":"# preflight: run as the fluentd user before starting fluentd\nBUFFER_DIR = '/var/log/fluent/buffer'\nDir.glob(\"#{BUFFER_DIR}/*.buf.meta\").each do |meta|\n  data = meta.sub(/\\.meta\\z/, '')\n  warn \"orphan meta: #{meta}\" unless File.exist?(data)\n  warn \"empty/truncated meta: #{meta}\" if File.size(meta) < 16\nend","typeGuard":null,"tryCatchPattern":"begin\n  chunk = Fluent::Plugin::Buffer::FileChunk.new(metadata, path, :staged)\nrescue Fluent::Plugin::Buffer::FileChunkError => e\n  log.error \"quarantining broken staged chunk #{path}: #{e.message}\"\n  File.rename(path, \"#{path}.broken\")\n  File.rename(\"#{path}.meta\", \"#{path}.meta.broken\") rescue nil\nend","preventionTips":["Stop fluentd with SIGTERM and let it flush instead of kill -9.","Monitor free space on the buffer partition and alert well before full.","Never share a buffer path between two fluentd instances or versions.","Drain queues (flush at shutdown) before upgrading fluentd.","Keep backups of the buffer directory taken while fluentd is stopped."],"tags":["fluentd","ruby","file-buffer","metadata-corruption","startup-failure"],"backgroundTag":"buffer-metadata-corruption","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}