phacility/phabricator · error · AphrontCharacterSetQueryException

Attempting to construct a query using a non-utf8 string when

Error message

Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%%B` conversion to escape binary strings data.

What it means

validateUTF8String() runs on every string bound into a query and throws AphrontCharacterSetQueryException when it is not valid UTF-8. The guard exists because MySQL's 3-byte utf8 charset silently truncates data (including 4-byte astral-plane characters such as emoji), which causes data loss and can even create security problems. Binary data must instead be passed through the %B conversion, which the message tells you to use.

Source

Thrown at src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:418

   * Force the next query to fail with a simulated error. This should be used
   * ONLY for unit tests.
   */
  public function simulateErrorOnNextQuery($error) {
    $this->nextError = $error;
    return $this;
  }

  /**
   * Check inserts for characters outside of the BMP. Even with the strictest
   * settings, MySQL will silently truncate data when it encounters these, which
   * can lead to data loss and security problems.
   */
  protected function validateUTF8String($string) {
    if (phutil_is_utf8($string)) {
      return;
    }

    throw new AphrontCharacterSetQueryException(
      pht(
        'Attempting to construct a query using a non-utf8 string when '.
        'utf8 is expected. Use the `%%B` conversion to escape binary '.
        'strings data.'));
  }

}

View on GitHub (pinned to 5720a38cfe)

Solutions

Bind binary values with %B (and lists of them with %LB) instead of %s
For text that should be text, sanitize first: phutil_utf8ize() strips/replaces invalid byte sequences
If you genuinely must store 4-byte characters as text, make sure the schema uses utf8mb4 rather than utf8
Do not silence the check or catch-and-continue: the truncation it prevents is silent data corruption

Example fix

// before: raw digest bytes through %s
queryfx($conn, 'INSERT INTO x (digest) VALUES (%s)', $raw_sha1_bytes);

// after: %B is the conversion for binary strings
queryfx($conn, 'INSERT INTO x (digest) VALUES (%B)', $raw_sha1_bytes);

Defensive patterns

Strategy: validation

Validate before calling

// Before binding a value, route it to the correct conversion:
if (!phutil_is_utf8($value)) {
  $pattern = '%B';   // binary: raw bytes
} else {
  $pattern = '%s';   // text: validated UTF-8
}
queryfx($conn, 'INSERT INTO t (v) VALUES ('.$pattern.')', $value);

Type guard

/** True when a value must be bound with %B rather than %s. */
function is_binary_string($value) {
  return is_string($value) && !phutil_is_utf8($value);
}

Try / catch

try {
  queryfx($conn, 'INSERT INTO t (v) VALUES (%s)', $value);
} catch (AphrontCharacterSetQueryException $ex) {
  // Do NOT retry with the same binding: sanitize or switch to %B.
  queryfx($conn, 'INSERT INTO t (v) VALUES (%B)', $value);
}

Prevention

Use %B (and %LB for lists) for every hash, HMAC, token, encrypted or serialized blob
Run phutil_utf8ize() over external text input before persisting it
Prefer utf8mb4 when 4-byte characters (emoji) must be stored as text
Never catch-and-ignore this exception: the alternative is silent truncation and data loss

When it happens

Trigger: Passing raw bytes through %s: SHA/ HMAC digests, encrypted or serialized blobs, compressed data, random tokens; user-submitted text containing 4-byte characters or truly invalid byte sequences; data imported from legacy latin1 columns without conversion.

Common situations: Storing file content hashes or cryptographic material by interpolating them as ordinary strings; copy-paste from word processors or mobile keyboards introducing astral characters; feeds and API payloads with mixed encodings entering the database layer.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/f167db9ea83c276f. Report an issue: GitHub.