fluent/fluentd · error · Fluent::ConfigError

Unexpected error: failed to read data from plugin storage…

Error message

Unexpected error: failed to read data from plugin storage file: '#{@path}'

What it means

This is the catch-all failure inside LocalStorage#configure's begin/rescue (storage_local.rb:95-97): any exception while opening or JSON.parse-ing the storage file — Errno::EACCES on open with wrong encoding flags, JSON::ParserError for malformed JSON, Encoding::UndefinedConversionError, or even the inner 'Invalid contents' ConfigError — is logged with the original error and re-raised as this Fluent::ConfigError. The actionable detail is in the preceding log.error line (path and the underlying error), not in this message.

Solutions

  1. Read the preceding 'failed to read data from plugin storage file' log entry to identify the real underlying error (path: and error: fields)
  2. If the file is corrupt JSON and the state is expendable, stop fluentd and delete/rename the file so a fresh one is created on next save
  3. Repair the JSON offline if the state matters: validate with ruby -rjson -e 'JSON.parse(File.read("storage.json"))' and fix until it parses to a Hash
  4. If the cause is an encoding error, rewrite the file as UTF-8 (the code opens it with 'r:utf-8:utf-8')

Example fix

# before
$ cat /var/log/fluent/storage.json
{"last_read": 1234   # truncated: missing closing brace
# => Unexpected error: failed to read data from plugin storage file: '...'

# after
$ cat /var/log/fluent/storage.json
{"last_read": 1234}
Defensive patterns

Strategy: try-catch

Validate before calling

require 'json'
path = '/var/log/fluent/storage.json'
begin
  JSON.parse(File.read(path, encoding: 'UTF-8')) if File.exist?(path) && !File.empty?(path)
rescue JSON::ParserError, EncodingError, SystemCallError => e
  raise "storage file unreadable, restore or delete it: #{e}"
end

Try / catch

begin
  plugin.configure(conf)
rescue Fluent::ConfigError => e
  if e.message =~ /failed to read data from plugin storage file/
    # the preceding log.error carries the original exception (parse/IO/encoding)
    # decide: repair JSON, or archive + delete the file to reset state
    exit 1
  end
  raise
end

Prevention

When it happens

Trigger: Storage file exists but contains malformed JSON (truncated by a crash mid-write, hand-edited and broken, or binary garbage); file exists but cannot be opened/read despite passing the readable? check (permissions changed racily, or an IO/encoding error); non-UTF-8 bytes that break the 'r:utf-8:utf-8' open.

Common situations: Power loss or SIGKILL leaving a partially written storage.json (no atomic rename landed); configuration-management tools templating the file badly; a path pointing at a directory's storage.json that was clobbered by logs; files transferred without binary-safe mode gaining a BOM or CRLF corruption.

Related errors

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/84eb7cbbaf717803. Report an issue: GitHub.

Appendix: source

Thrown at lib/fluent/plugin/storage_local.rb:97

          end
        end

        if !@on_memory
          dir = File.dirname(@path)
          FileUtils.mkdir_p(dir, mode: @dir_mode) unless Dir.exist?(dir)
          if File.exist?(@path)
            raise Fluent::ConfigError, "Plugin storage path '#{@path}' is not readable/writable" unless File.readable?(@path) && File.writable?(@path)
            begin
              data = File.open(@path, 'r:utf-8:utf-8') { |io| io.read }
              if data.empty?
                log.warn "detect empty plugin storage file during startup. Ignored: #{@path}"
                return
              end
              data = JSON.parse(data, Fluent::DEFAULT_JSON_PARSE_OPTIONS)
              raise Fluent::ConfigError, "Invalid contents (not object) in plugin storage file: '#{@path}'" unless data.is_a?(Hash)
            rescue => e
              log.error "failed to read data from plugin storage file", path: @path, error: e
              raise Fluent::ConfigError, "Unexpected error: failed to read data from plugin storage file: '#{@path}'"
            end
          else
            raise Fluent::ConfigError, "Directory is not writable for plugin storage file '#{@path}'" unless File.stat(dir).writable?
          end
        end
      end

      def multi_workers_ready?
        unless @multi_workers_available
          log.error "local plugin storage with multi workers should be configured to use directory 'path', or system root_dir and plugin id"
        end
        @multi_workers_available
      end

      def load
        return if @on_memory
        return unless File.exist?(@path)
        begin

View on GitHub (pinned to dd45c6e18d)