###WARNING: Encountered abnormal bytes while converting inp

Error message

###WARNING: Encountered abnormal bytes while converting input stream to target encoding: %s

What it means

Emitted from ucbuf_fillucbuf right after ucnv_toUnicode fails. The converter was configured with UCNV_TO_U_CALLBACK_STOP, so any byte sequence illegal for the target encoding halts conversion and sets a failure status. When showWarning is true this stderr line reports the failure (via u_errorName) together with pre/post context around the offending bytes, then the code resets the converter and switches to a substitution callback so reading can continue lossily.

Source

Thrown at deps/icu-small/source/tools/toolutil/ucbuf.cpp:238

        source = cbuf;
        sourceLimit = source + inputRead;
        ucnv_toUnicode(buf->conv,&target,target+(buf->bufCapacity-offset),
                        &source,sourceLimit,nullptr,
                        static_cast<UBool>(buf->remaining == 0), error);

        if(U_FAILURE(*error)){
            char context[CONTEXT_LEN+1];
            char preContext[CONTEXT_LEN+1];
            char postContext[CONTEXT_LEN+1];
            int8_t len = CONTEXT_LEN;
            int32_t start=0;
            int32_t stop =0;
            int32_t pos =0;
            /* use erro1 to preserve the error code */
            UErrorCode error1 =U_ZERO_ERROR;
            
            if( buf->showWarning==true){
                fprintf(stderr,"\n###WARNING: Encountered abnormal bytes while"
                               " converting input stream to target encoding: %s\n",
                               u_errorName(*error));
            }


            /* now get the context chars */
            ucnv_getInvalidChars(buf->conv,context,&len,&error1);
            context[len]= 0 ; /* null terminate the buffer */

            pos = static_cast<int32_t>(source - cbuf - len);

            /* for pre-context */
            start = (pos <=CONTEXT_LEN)? 0 : (pos - (CONTEXT_LEN-1));
            stop  = pos-len;

            memcpy(preContext,cbuf+start,stop-start);
            /* null terminate the buffer */
            preContext[stop-start] = 0;

View on GitHub (pinned to 1b2de5e052)

Solutions

Re-save the input file in the encoding ICU detected or was told to use; verify with `file -i` or `uchardet` and confirm the byte at the reported Pre/Post-context offset.
Pass the correct codepage to ucbuf_open via *cp (or fix/remove a mismatched BOM) so autodetect and explicit name agree.
If lossy conversion is acceptable, keep showWarning=true and rely on the built-in substitution callback that resumes after this warning.
Sanitize/repair the file with iconv (-c to drop, //IGNORE to substitute) before feeding it to the tool.

Example fix

// before: file is Latin-1 but opened with no cp, BOM missing
UCHARBUF* b = ucbuf_open(path, &cp, /*showWarning=*/true, /*buffered=*/false, &status);
// after: force the real encoding so ucnv_toUnicode never hits illegal bytes
const char* cp = "windows-1252";
UCHARBUF* b = ucbuf_open(path, &cp, /*showWarning=*/true, /*buffered=*/false, &status);

Defensive patterns

Strategy: validation

Validate before calling

// Validate the file's bytes against the target encoding before ucbuf_open.
#include <unicode/ucnv.h>
#include <unicode/uchar.h>
bool bytes_valid_for(const char* path, const char* cp) {
    UErrorCode err = U_ZERO_ERROR;
    UConverter* c = ucnv_open(cp, &err);
    if (U_FAILURE(err) || !c) return false;
    // read whole file into `buf`, `n`
    // (elided: fopen/fread)
    const char* src = buf;
    const char* lim = buf + n;
    UChar out[256];
    bool ok = true;
    while (src < lim) {
        UChar* tgt = out;
        UErrorCode e = U_ZERO_ERROR;
        ucnv_toUnicode(c, &tgt, out + 256, &src, lim, nullptr, true, &e);
        if (U_FAILURE(e)) { ok = false; break; }
    }
    ucnv_close(c);
    return ok;
}

Try / catch

// ICU uses status codes, not exceptions. Check after each fill/read:
UErrorCode status = U_ZERO_ERROR;
const char* cp = "UTF-8";
UCHARBUF* buf = ucbuf_open(path, &cp, /*showWarning=*/true, false, &status);
if (U_FAILURE(status)) { /* handle open failure */ }
ucbuf_fillucbuf(buf, &status);
if (status == U_INVALID_CHAR_FOUND || status == U_TRUNCATED_CHAR_FOUND ||
    status == U_ILLEGAL_CHAR_FOUND) {
    // illegal input bytes for the encoding -- the warning has already been
    // printed; decide whether to accept lossy substitution or abort.
}

Prevention

Standardize all data inputs on UTF-8 with a BOM and reject files that fail `file -i` / `uchardet`.
Never leave *cp null when calling ucbuf_open on untrusted files; always pass an explicit canonical encoding.
Run builds with showWarning=true in CI so encoding issues surface instead of silently substituting.
Canonicalize inputs through iconv to UTF-8 before feeding them to ICU toolutil.

When it happens

Trigger: Calling ucbuf_open then ucbuf_fillucbuf (directly or via the ICU tools) on a file whose bytes do not match the detected-BOM encoding or the explicitly passed *cp codepage: a file opened as UTF-8 that contains a stray Latin-1/0x80-0xFF byte, a UTF-16 file read as UTF-8, truncated multibyte sequences, or any byte illegal under ucnv_toUnicode with the STOP callback.

Common situations: Resource/data .txt files saved by an editor with a different encoding than declared; files transferred through tools that re-encode or strip bytes; a wrong BOM; concatenation of mixed-encoding fragments; locale data regenerated on a machine whose default encoding differs from the data's.

Related errors

AI-assisted analysis of nodejs/node@1b2de5e052 (2026-08-13). Data as JSON: /api/errors/fe38404434a4e406. Report an issue: GitHub.