{"record":{"id":"f167db9ea83c276f","repo":"phacility/phabricator","slug":"attempting-to-construct-a-query-using-a-non-utf8-s","errorCode":null,"errorMessage":"Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%%B` conversion to escape binary strings data.","messagePattern":"Attempting to construct a query using a non-utf8 string when utf8 is expected\\. Use the `%%B` conversion to escape binary strings data\\.","errorType":"exception","errorClass":"AphrontCharacterSetQueryException","httpStatus":null,"severity":"error","filePath":"src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php","lineNumber":418,"sourceCode":"   * Force the next query to fail with a simulated error. This should be used\n   * ONLY for unit tests.\n   */\n  public function simulateErrorOnNextQuery($error) {\n    $this->nextError = $error;\n    return $this;\n  }\n\n  /**\n   * Check inserts for characters outside of the BMP. Even with the strictest\n   * settings, MySQL will silently truncate data when it encounters these, which\n   * can lead to data loss and security problems.\n   */\n  protected function validateUTF8String($string) {\n    if (phutil_is_utf8($string)) {\n      return;\n    }\n\n    throw new AphrontCharacterSetQueryException(\n      pht(\n        'Attempting to construct a query using a non-utf8 string when '.\n        'utf8 is expected. Use the `%%B` conversion to escape binary '.\n        'strings data.'));\n  }\n\n}\n","sourceCodeStart":400,"sourceCodeEnd":426,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php#L400-L426","documentation":"validateUTF8String() runs on every string bound into a query and throws AphrontCharacterSetQueryException when it is not valid UTF-8. The guard exists because MySQL's 3-byte utf8 charset silently truncates data (including 4-byte astral-plane characters such as emoji), which causes data loss and can even create security problems. Binary data must instead be passed through the %B conversion, which the message tells you to use.","triggerScenarios":"Passing raw bytes through %s: SHA/ HMAC digests, encrypted or serialized blobs, compressed data, random tokens; user-submitted text containing 4-byte characters or truly invalid byte sequences; data imported from legacy latin1 columns without conversion.","commonSituations":"Storing file content hashes or cryptographic material by interpolating them as ordinary strings; copy-paste from word processors or mobile keyboards introducing astral characters; feeds and API payloads with mixed encodings entering the database layer.","solutions":["Bind binary values with %B (and lists of them with %LB) instead of %s","For text that should be text, sanitize first: phutil_utf8ize() strips/replaces invalid byte sequences","If you genuinely must store 4-byte characters as text, make sure the schema uses utf8mb4 rather than utf8","Do not silence the check or catch-and-continue: the truncation it prevents is silent data corruption"],"exampleFix":"// before: raw digest bytes through %s\nqueryfx($conn, 'INSERT INTO x (digest) VALUES (%s)', $raw_sha1_bytes);\n\n// after: %B is the conversion for binary strings\nqueryfx($conn, 'INSERT INTO x (digest) VALUES (%B)', $raw_sha1_bytes);","handlingStrategy":"validation","validationCode":"// Before binding a value, route it to the correct conversion:\nif (!phutil_is_utf8($value)) {\n  $pattern = '%B';   // binary: raw bytes\n} else {\n  $pattern = '%s';   // text: validated UTF-8\n}\nqueryfx($conn, 'INSERT INTO t (v) VALUES ('.$pattern.')', $value);","typeGuard":"/** True when a value must be bound with %B rather than %s. */\nfunction is_binary_string($value) {\n  return is_string($value) && !phutil_is_utf8($value);\n}","tryCatchPattern":"try {\n  queryfx($conn, 'INSERT INTO t (v) VALUES (%s)', $value);\n} catch (AphrontCharacterSetQueryException $ex) {\n  // Do NOT retry with the same binding: sanitize or switch to %B.\n  queryfx($conn, 'INSERT INTO t (v) VALUES (%B)', $value);\n}","preventionTips":["Use %B (and %LB for lists) for every hash, HMAC, token, encrypted or serialized blob","Run phutil_utf8ize() over external text input before persisting it","Prefer utf8mb4 when 4-byte characters (emoji) must be stored as text","Never catch-and-ignore this exception: the alternative is silent truncation and data loss"],"tags":["mysql","utf8","charset","binary-data","data-loss","validation"],"backgroundTag":"character-encoding-mismatch","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}