fluent/fluentd · error · Fluent::InvalidLockDirectory

can't acquire lock because FLUENTD_LOCK_DIR isn't set

Error message

can't acquire lock because FLUENTD_LOCK_DIR isn't set

What it means

acquire_worker_lock serializes plugin work across worker processes using flock files under a lock directory taken from ENV['FLUENTD_LOCK_DIR'] (read in Fluent::Plugin::Base#initialize). The supervisor exports that env var when it boots workers (supervisor.rb); core users include out_file append mode and output flush locking (output.rb, out_file.rb). When a plugin calls the lock outside a supervised fluentd process — unit tests, standalone embedding, or an environment where the variable was scrubbed — the nil lock dir raises Fluent::InvalidLockDirectory.

Source

Thrown at lib/fluent/plugin/base.rb:84

        self
      end

      def multi_workers_ready?
        true
      end

      LOCK_FILE_BUCKETS = 65536

      def get_lock_path(name)
        # The mapping from a name to a bucket MUST be identical across worker processes.
        # Ruby's String#hash is randomly seeded per process and MUST NOT be used here.
        bucket = Zlib.crc32(name.to_s) % LOCK_FILE_BUCKETS
        File.join(@fluentd_lock_dir, "fluentd-bucket-#{bucket}.lock")
      end

      def acquire_worker_lock(name)
        if @fluentd_lock_dir.nil?
          raise InvalidLockDirectory, "can't acquire lock because FLUENTD_LOCK_DIR isn't set"
        end
        lock_path = get_lock_path(name)
        File.open(lock_path, "w") do |f|
          f.flock(File::LOCK_EX)
          yield
        end
        # Update access time to prevent tmpwatch from deleting a lock file.
        FileUtils.touch(lock_path)
      end

      def string_safe_encoding(str)
        unless str.valid_encoding?
          str = str.scrub('?')
          log.info "invalid byte sequence is replaced in `#{str}`" if self.respond_to?(:log)
        end
        yield str
      end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Set FLUENTD_LOCK_DIR to a writable directory before the code path runs: export FLUENTD_LOCK_DIR=$(mktemp -d)
2. In tests, set ENV['FLUENTD_LOCK_DIR'] = Dir.mktmpdir in setup (and clean up in teardown)
3. Run the plugin under the normal fluentd supervisor, which exports the variable itself
4. For containerized deployments, ensure the env var survives into the worker process (do not scrub the environment)

Example fix

# before
# spec: plugin calls acquire_worker_lock -> Fluent::InvalidLockDirectory
it 'appends' do
  driver.configure(fluent_config)
end

# after
around do |ex|
  ENV['FLUENTD_LOCK_DIR'] = Dir.mktmpdir
  ex.run
  ENV.delete('FLUENTD_LOCK_DIR')
end
it 'appends' do
  driver.configure(fluent_config)
end

lock_dir = ENV['FLUENTD_LOCK_DIR']
raise 'FLUENTD_LOCK_DIR unset; cannot use worker locks' if lock_dir.nil?
require 'fileutils'
FileUtils.mkdir_p(lock_dir) unless Dir.exist?(lock_dir)

begin
  plugin.acquire_worker_lock('name') { work }
rescue Fluent::InvalidLockDirectory
  ENV['FLUENTD_LOCK_DIR'] = Dir.mktmpdir
  retry
end

When it happens

Trigger: Running plugin unit tests (e.g. out_file with append) without setting FLUENTD_LOCK_DIR; embedding Fluent::Engine in a custom process instead of the fluentd supervisor; systemd/docker hardening (EnvironmentFile omission, env clear) dropping the variable inherited from the supervisor.

Common situations: RSpec failures appearing only when a locking code path is exercised; CI running plugins in isolation; custom launchers that exec worker code directly rather than through fluentd's supervisor.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• invalid_params
• unknown_key
• Unknown #{@kind} plugin '#{type}'. Run 'gem search -rd fluen
• Missing '@type' parameter on <#{e.name}> directive
• '#{capability}' is not valid capability. Valid Capabilities

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/5ce1842b2d4ebc80. Report an issue: GitHub.