fluent/fluentd · error · Fluent::ConfigError

Duplicate key: #{e.key}

Error message

Duplicate key: #{e.key}

What it means

After the deprecated regexpN lines are parsed, configure folds in the modern <regexp> section directives, checking each against the regexp_and_conditions hash. If a <regexp> section's key equals a key already supplied via regexpN (or via an earlier duplicate section fold-in), configure raises Fluent::ConfigError 'Duplicate key'.

Source

Thrown at lib/fluent/plugin/filter_grep.rb:116

        key, regexp = conf["regexp#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "regexp#{i} does not contain 2 parameters" unless regexp
        raise Fluent::ConfigError, "regexp#{i} contains a duplicated key, #{key}" if regexp_and_conditions[key]
        regexp_and_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(regexp))
      end

      (1..REGEXP_MAX_NUM).each do |i|
        next unless conf["exclude#{i}"]
        key, exclude = conf["exclude#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "exclude#{i} does not contain 2 parameters" unless exclude
        raise Fluent::ConfigError, "exclude#{i} contains a duplicated key, #{key}" if exclude_or_conditions[key]
        exclude_or_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(exclude))
      end

      if @regexps.size > 1
        log.info "Top level multiple <regexp> is interpreted as 'and' condition"
      end
      @regexps.each do |e|
        raise Fluent::ConfigError, "Duplicate key: #{e.key}" if regexp_and_conditions.key?(e.key)
        regexp_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
      end

      if @excludes.size > 1
        log.info "Top level multiple <exclude> is interpreted as 'or' condition"
      end
      @excludes.each do |e|
        raise Fluent::ConfigError, "Duplicate key: #{e.key}" if exclude_or_conditions.key?(e.key)
        exclude_or_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
      end

      @and_conditions.each do |and_condition|
        if !and_condition.regexps.empty? && !and_condition.excludes.empty?
          raise Fluent::ConfigError, "Do not specify both <regexp> and <exclude> in <and>"
        end
        and_condition.regexps.each do |e|
          raise Fluent::ConfigError, "Duplicate key in <and>: #{e.key}" if regexp_and_conditions.key?(e.key)
          regexp_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Delete the deprecated regexpN line for that key, keeping the <regexp> section
2. Or remove the <regexp> section if the old line is intentional (then plan migration)
3. Grep the config for the duplicated field name to find both occurrences
4. Validate with fluentd --dry-run

Example fix

# before
<filter app.**>
  @type grep
  regexp1 message /error/
  <regexp>
    key message
    pattern /fatal/
  </regexp>
</filter>

# after
<filter app.**>
  @type grep
  <regexp>
    key message
    pattern /error/
  </regexp>
  <regexp>
    key message
    pattern /fatal/
  </regexp>
</filter>

old_keys = (1..20).filter_map { |i| conf["regexp#{i}"]&.split(/ /, 2)&.first }
new_keys = grep_filter_conf.sections(:regexp).map { |s| s['key'] }
overlap = old_keys & new_keys
abort "field(s) declared in both regexpN and <regexp>: #{overlap.join(', ')}" unless overlap.empty?

begin
  filter.configure(conf)
rescue Fluent::ConfigError => e
  abort "grep key collision: #{e.message}" if e.message.start_with?('Duplicate key:')
  raise
end

When it happens

Trigger: A config mixing old and new syntax on the same field, e.g. regexp1 message /a/ together with <regexp> key message ... </regexp>; both register the key 'message' in the AND-condition map.

Common situations: Incremental migration where the new <regexp> section was added but the old regexpN line was left in place; config management tools merging old and new stanzas.

Related errors

• regexp#{i} contains a duplicated key, #{key}
• exclude#{i} contains a duplicated key, #{key}
• regexp#{i} does not contain 2 parameters
• exclude#{i} does not contain 2 parameters
• Do not specify both <regexp> and <exclude> in <and>

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/b11d9f57158d72d6. Report an issue: GitHub.