fluent/fluentd · error · Fluent::ConfigError

Duplicate key in <or>: #{e.key}

Error message

Duplicate key in <or>: #{e.key}

What it means

The grep filter builds lookup hashes keyed by record field; while iterating every <or> block it raises Fluent::ConfigError 'Duplicate key in <or>: <key>' when a regexp key already exists in regexp_or_conditions. Because the hash accumulates across all <or> sections, the duplicate can be two <regexp> entries for the same key inside one <or> or the same key reused in different <or> blocks. This is a load-time configuration error: fluentd fails to start.

Source

Thrown at lib/fluent/plugin/filter_grep.rb:147

        if !and_condition.regexps.empty? && !and_condition.excludes.empty?
          raise Fluent::ConfigError, "Do not specify both <regexp> and <exclude> in <and>"
        end
        and_condition.regexps.each do |e|
          raise Fluent::ConfigError, "Duplicate key in <and>: #{e.key}" if regexp_and_conditions.key?(e.key)
          regexp_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
        end
        and_condition.excludes.each do |e|
          raise Fluent::ConfigError, "Duplicate key in <and>: #{e.key}" if exclude_and_conditions.key?(e.key)
          exclude_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
        end
      end

      @or_conditions.each do |or_condition|
        if !or_condition.regexps.empty? && !or_condition.excludes.empty?
          raise Fluent::ConfigError, "Do not specify both <regexp> and <exclude> in <or>"
        end
        or_condition.regexps.each do |e|
          raise Fluent::ConfigError, "Duplicate key in <or>: #{e.key}" if regexp_or_conditions.key?(e.key)
          regexp_or_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
        end
        or_condition.excludes.each do |e|
          raise Fluent::ConfigError, "Duplicate key in <or>: #{e.key}" if exclude_or_conditions.key?(e.key)
          exclude_or_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
        end
      end

      @_regexp_and_conditions = regexp_and_conditions.values unless regexp_and_conditions.empty?
      @_exclude_and_conditions = exclude_and_conditions.values unless exclude_and_conditions.empty?
      @_regexp_or_conditions = regexp_or_conditions.values unless regexp_or_conditions.empty?
      @_exclude_or_conditions = exclude_or_conditions.values unless exclude_or_conditions.empty?
    end

    def filter(tag, time, record)
      begin
        if @_regexp_and_conditions && @_regexp_and_conditions.any? { |expression| !expression.match?(record) }
          return nil

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Deduplicate: keep one <regexp> entry per field key across all <or> blocks.
2. Combine multiple patterns for one key into a single regex with alternation, e.g. regexp1 message (timeout|refused).
3. Run fluentd --dry-run -c /etc/fluent/fluent.conf in CI to catch config errors before deploy.

Example fix

# before
<filter app.**>
  @type grep
  <or>
    regexp1 message error
    regexp1 message ERROR
  </or>
</filter>
# after
<filter app.**>
  @type grep
  <or>
    regexp1 message (?i)error
  </or>
</filter>

# CI/config gate: fail before deploy on duplicate regexp keys across <or> blocks
# (run: fluentd --dry-run -c /etc/fluent/fluent.conf && echo OK)
# or scan the config text:
keys = conf.scan(/^\s*regexp\d+\s+(\S+)/).flatten
counts = Hash.new(0); keys.each { |k| counts[k] += 1 }
abort "duplicate regexp keys: #{counts.select { |_k, c| c > 1 }.keys}" unless counts.values.all? { |c| c == 1 }

begin
  Fluent::Plugin.new_filter('grep').configure(conf)
rescue Fluent::ConfigError => e
  abort "grep filter config rejected: #{e.message}" # duplicate key in <or>: ...
end

When it happens

Trigger: Two or more <regexp> stanzas in <or> using the same field key (e.g. regexp1 message ... twice), or the same key appearing in separate <or> blocks; the second insertion triggers the check before the Expression is created.

Common situations: Copy-pasted regexp lines where only the pattern differs; teams adding another <or> block for the same field; refactors that merge <and> content into <or> without deduping keys.

Related errors

• regexp#{i} contains a duplicated key, #{key}
• exclude#{i} contains a duplicated key, #{key}
• Duplicate key: #{e.key}
• `renew_record` must be true to use `keep_keys`
• empty value is specified in fields parameter

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/b4a2f574828fce62. Report an issue: GitHub.