fluent/fluentd · error · Fluent::ConfigError

Unexpected error: failed to read data from plugin storage fi

Error message

Unexpected error: failed to read data from plugin storage file: '#{@path}'

What it means

This is the catch-all failure inside LocalStorage#configure's begin/rescue (storage_local.rb:95-97): any exception while opening or JSON.parse-ing the storage file — Errno::EACCES on open with wrong encoding flags, JSON::ParserError for malformed JSON, Encoding::UndefinedConversionError, or even the inner 'Invalid contents' ConfigError — is logged with the original error and re-raised as this Fluent::ConfigError. The actionable detail is in the preceding log.error line (path and the underlying error), not in this message.

Source

Thrown at lib/fluent/plugin/storage_local.rb:97

          end
        end

        if !@on_memory
          dir = File.dirname(@path)
          FileUtils.mkdir_p(dir, mode: @dir_mode) unless Dir.exist?(dir)
          if File.exist?(@path)
            raise Fluent::ConfigError, "Plugin storage path '#{@path}' is not readable/writable" unless File.readable?(@path) && File.writable?(@path)
            begin
              data = File.open(@path, 'r:utf-8:utf-8') { |io| io.read }
              if data.empty?
                log.warn "detect empty plugin storage file during startup. Ignored: #{@path}"
                return
              end
              data = JSON.parse(data, Fluent::DEFAULT_JSON_PARSE_OPTIONS)
              raise Fluent::ConfigError, "Invalid contents (not object) in plugin storage file: '#{@path}'" unless data.is_a?(Hash)
            rescue => e
              log.error "failed to read data from plugin storage file", path: @path, error: e
              raise Fluent::ConfigError, "Unexpected error: failed to read data from plugin storage file: '#{@path}'"
            end
          else
            raise Fluent::ConfigError, "Directory is not writable for plugin storage file '#{@path}'" unless File.stat(dir).writable?
          end
        end
      end

      def multi_workers_ready?
        unless @multi_workers_available
          log.error "local plugin storage with multi workers should be configured to use directory 'path', or system root_dir and plugin id"
        end
        @multi_workers_available
      end

      def load
        return if @on_memory
        return unless File.exist?(@path)
        begin

View on GitHub (pinned to dd45c6e18d)

Solutions

  1. Read the preceding 'failed to read data from plugin storage file' log entry to identify the real underlying error (path: and error: fields)
  2. If the file is corrupt JSON and the state is expendable, stop fluentd and delete/rename the file so a fresh one is created on next save
  3. Repair the JSON offline if the state matters: validate with ruby -rjson -e 'JSON.parse(File.read("storage.json"))' and fix until it parses to a Hash
  4. If the cause is an encoding error, rewrite the file as UTF-8 (the code opens it with 'r:utf-8:utf-8')

Example fix

# before
$ cat /var/log/fluent/storage.json
{"last_read": 1234   # truncated: missing closing brace
# => Unexpected error: failed to read data from plugin storage file: '...'

# after
$ cat /var/log/fluent/storage.json
{"last_read": 1234}
Defensive patterns

Strategy: try-catch

Validate before calling

require 'json'
path = '/var/log/fluent/storage.json'
begin
  JSON.parse(File.read(path, encoding: 'UTF-8')) if File.exist?(path) && !File.empty?(path)
rescue JSON::ParserError, EncodingError, SystemCallError => e
  raise "storage file unreadable, restore or delete it: #{e}"
end

Try / catch

begin
  plugin.configure(conf)
rescue Fluent::ConfigError => e
  if e.message =~ /failed to read data from plugin storage file/
    # the preceding log.error carries the original exception (parse/IO/encoding)
    # decide: repair JSON, or archive + delete the file to reset state
    exit 1
  end
  raise
end

Prevention

When it happens

Trigger: Storage file exists but contains malformed JSON (truncated by a crash mid-write, hand-edited and broken, or binary garbage); file exists but cannot be opened/read despite passing the readable? check (permissions changed racily, or an IO/encoding error); non-UTF-8 bytes that break the 'r:utf-8:utf-8' open.

Common situations: Power loss or SIGKILL leaving a partially written storage.json (no atomic rename landed); configuration-management tools templating the file badly; a path pointing at a directory's storage.json that was clobbered by logs; files transferred without binary-safe mode gaining a BOM or CRLF corruption.

Related errors

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/84eb7cbbaf717803. Report an issue: GitHub.