What causes "experiment '#{name}' observations mismatched" in github/scientist?

Fix the underlying behavioral difference so candidate and control return equivalent results.

github/scientist · error · Scientist::Experiment::MismatchError

experiment '# ' observations mismatched

Error message

experiment '#{name}' observations mismatched

What it means

Scientist::Experiment::MismatchError is raised by Experiment#run when `raise_on_mismatches?` is enabled and the published result has mismatched observations (candidate returned a different value, or raised a different error, than control per the configured comparators). If a custom error class was set via `raise_with`, that class is instantiated and raised instead.

Solutions

Fix the underlying behavioral difference so candidate and control return equivalent results.
Relax or correct the `compare` / `compare_errors` comparator if the mismatch is a false positive.
Disable raising on mismatches (`raise_on_mismatches { false }`) and inspect published results instead.
Rescue Scientist::Experiment::MismatchError (or your raise_with class) around run if divergence is tolerable.

Example fix

# before
experiment.raise_on_mismatches { true }
experiment.use { user.name }
experiment.try { user.full_name } # mismatch raised in run

# after
experiment.compare { |a, b| a.to_s == b.to_s }
experiment.raise_on_mismatches { Rails.env.test? }
experiment.use { user.name }
experiment.try { user.full_name }

Defensive patterns

Strategy: try-catch

Try / catch

begin
  value = experiment.run
rescue Scientist::Experiment::MismatchError => e
  metrics.increment("scientist.mismatch")
  e.result.control.value # fall back to control behavior
end

Prevention

Enable raise_on_mismatches only in test/CI; keep it off in production paths.
Write explicit `compare` lambdas covering known non-determinism (time, ids, ordering).
Use `compare_errors` so differing-but-acceptable exception classes don't count as mismatches.
Watch published results in your publisher to catch drift before it raises.

When it happens

Trigger: Calling `experiment.run` with raise_on_mismatches enabled (via `raise_on_mismatches { true }`) when the candidate block returns a value not equivalent to control, or when a `compare`/`compare_errors` lambda reports inequality. Also raised when the custom `raise_with` class is instantiated on a mismatch.

Common situations: Test/CI environments that enable raise_on_mismatches to catch divergent behavior; candidate and control genuinely diverging after a code change; an overly strict comparator (timestamps, object identity) causing false mismatches; candidate code upgraded to a different return shape.

Related errors

AI-assisted analysis of github/scientist@504a396e98 (2026-09-13). Data as JSON: /api/errors/e2e7309526c1c974. Report an issue: GitHub.

Appendix: source

Thrown at lib/scientist/experiment.rb:257

    end

    result = generate_result(name)

    if @_scientist_after_run
      @_scientist_after_run.call(result)
    end

    begin
      publish(result)
    rescue StandardError => ex
      raised :publish, ex
    end

    if raise_on_mismatches? && result.mismatched?
      if @_scientist_custom_mismatch_error
        raise @_scientist_custom_mismatch_error.new(self.name, result)
      else
        raise MismatchError.new(self.name, result)
      end
    end

    control = result.control
    raise control.exception if control.raised?
    control.value
  end

  # Define a block that determines whether or not the experiment should run.
  def run_if(&block)
    @_scientist_run_if_block = block
  end

  # Internal: does a run_if block allow the experiment to run?
  #
  # Rescues and reports exceptions in a run_if block if they occur.
  def run_if_block_allows?
    (@_scientist_run_if_block ? @_scientist_run_if_block.call : true)

View on GitHub (pinned to 504a396e98)