urbanadventurer/WhatWeb · warning

Can't convert to UTF-8 #

Error message

Can't convert to UTF-8 #{e}

What it means

A generic RuntimeError raised in lib/helper.rb when the UTF-8 sanitization helper (force_encoding + scrub) fails for any reason; the original exception is re-raised with its message appended. It signals that a string pulled from a target response could not be normalized to valid UTF-8 for further matching/output.

Solutions

1. Check the actual encoding of the input string (str.encoding) and convert with encode('UTF-8', invalid: :replace, undef: :replace) instead of force_encoding
2. Inspect the appended #{e} message in the raised error to identify the underlying failure (frozen string, bad byte sequence, etc.)
3. Call .b or .dup before force_encoding on strings you do not own
4. Pre-filter obviously binary payloads before handing them to the helper

Example fix

# before
raise "Can't convert to UTF-8 #{e}"
# after: convert safely instead of force_encoding
str.dup.force_encoding('ASCII-8BIT').encode('UTF-8', invalid: :replace, undef: :replace)

def safe_utf8?(str)
  str.is_a?(String) && str.dup.force_encoding('UTF-8').valid_encoding?
end

def as_utf8(str)
  return nil unless str.is_a?(String)
  str.dup.force_encoding('UTF-8').scrub
end

begin
  text = EncodingHelper.scrub(raw)
rescue RuntimeError => e
  logger.warn("UTF-8 conversion failed: #{e.message}")
  text = raw.b.force_encoding('UTF-8', invalid: :replace, undef: :replace) rescue ''
end

When it happens

Trigger: Passing binary or non-UTF-8 encoded data (e.g. responses in another encoding, gzip bodies decoded incorrectly, raw bytes from sockets) into the helper's encode/scrub method where force_encoding or scrub itself raises (e.g. Encoding::UndefinedConversionError or a frozen-string/argument issue not covered by the dup branch).

Common situations: Scanning servers returning Latin-1, Shift-JIS, or otherwise mislabeled content; plugin logic feeding raw binary blobs (images, compressed data) into text matching; Ruby frozen string literal magic comments interacting with the frozen? branch.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

AI-assisted analysis of urbanadventurer/WhatWeb@d279d93042 (2026-09-15). Data as JSON: /api/errors/888159fb11e378cc. Report an issue: GitHub.

Appendix: source

Thrown at lib/helper.rb:47

    elsif obj.class == Array
      obj.each do |x|
        utf8_elements!(x)
      end
    elsif obj.class == String
      convert_to_utf8(obj)
    end
  end

  # Converts a string to UTF-8
  def self.convert_to_utf8(str)
    begin
      if (str.frozen?)
        str.dup.force_encoding("UTF-8").scrub
      else
        str.force_encoding("UTF-8").scrub
      end
    rescue => e
      raise "Can't convert to UTF-8 #{e}"
    end
  end

  #
  # Takes an integer of certainty (between 1 - 100)
  #
  # returns String a word representing the certainty
  #
  def self.certainty_to_words(certainty)
    case certainty
    when 0..49
      'maybe'
    when 50..99
      'probably'
    when 100
      'certain'
    end
  end

View on GitHub (pinned to d279d93042)