puppetlabs/puppet · error · Timeout::Error

Timeout waiting for exclusive lock on %{path}

Error message

Timeout waiting for exclusive lock on %{path}

What it means

Puppet's atomic write helper (FileImpl#replace_file) opens the target and tries flock LOCK_EX|LOCK_NB; on contention it sleeps with exponential backoff while a timeout budget lasts, then raises Timeout::Error. Another process is holding an exclusive lock on the same file, typically concurrent Puppet runs or a stuck process sharing the same vardir/file.

Source

Thrown at lib/puppet/file_system/file_impl.rb:68

  end

  def exclusive_open(path, mode, options = 'r', timeout = 300, &block)
    wait = 0.001 + (Kernel.rand / 1000)
    written = false
    until written
      ::File.open(path, options, mode) do |rf|
        if rf.flock(::File::LOCK_EX | ::File::LOCK_NB)
          Puppet.debug { _("Locked '%{path}'") % { path: path } }
          yield rf
          written = true
          Puppet.debug { _("Unlocked '%{path}'") % { path: path } }
        else
          Puppet.debug { "Failed to lock '%s' retrying in %.2f milliseconds" % [path, wait * 1000] }
          sleep wait
          timeout -= wait
          wait *= 2
          if timeout < 0
            raise Timeout::Error, _("Timeout waiting for exclusive lock on %{path}") % { path: path }
          end
        end
      end
    end
  end

  def each_line(path, &block)
    ::File.open(path) do |f|
      f.each_line do |line|
        yield line
      end
    end
  end

  def read(path, opts = {})
    path.read(**opts)
  end

View on GitHub (pinned to e227c27540)

Solutions

1. Find the lock holder: `lsof /path/to/file` or `fuser -v`, then stop that process cleanly
2. Prevent overlap: ensure one agent per vardir (unique ssldir/vardir per container), fix runinterval/cron so runs cannot collide
3. If the holder is dead but state is stale, verify with lsof that no fd remains, then retry the operation
4. In custom code using replace_file, catch Timeout::Error and retry with backoff, or serialize access yourself

Example fix

# before
Puppet::FileSystem.replace_file(path, 0o644) { |f| f.write(data) } # may raise Timeout::Error

# after
attempts = 0
begin
  attempts += 1
  Puppet::FileSystem.replace_file(path, 0o644) { |f| f.write(data) }
rescue Timeout::Error
  raise if attempts >= 3
  sleep(2**attempts)
  retry
end

# Before writing, confirm no other process holds the file
def locked_by?(path)
  !system("lsof -- #{Shellwords.escape(path)} >/dev/null 2>&1")
end

attempts = 0
begin
  Puppet::FileSystem.replace_file(path, mode) { |f| f.write(data) }
rescue Timeout::Error
  attempts += 1
  raise if attempts > 3
  sleep(2**attempts)
  retry
end

When it happens

Trigger: Two puppet agent processes sharing one vardir (containers with a shared volume, or a stray agent plus a cron run); a hung ruby/puppetserver process never releasing flock; Puppet::FileSystem.replace_file called concurrently by custom face/tool code on the same path.

Common situations: Overlapping puppet agent schedules (runinterval shorter than run time plus manual `puppet agent --test`); Docker images running agents against a mounted vardir; orphaned processes after a crashed run.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Is a directory: %{directory}
• Operation not permitted
• Is a directory: %{directory}
• Timed out waiting for #{@resource[:name]} to transition stat
• Timed out waiting for #{@resource[:name]} to transition stat

AI-assisted analysis of puppetlabs/puppet@e227c27540 (2026-08-21). Data as JSON: /api/errors/7c5a44afd4fc38ec. Report an issue: GitHub.