{"record":{"id":"fe38404434a4e406","repo":"nodejs/node","slug":"warning-encountered-abnormal-bytes-while-conv","errorCode":null,"errorMessage":"\n###WARNING: Encountered abnormal bytes while converting input stream to target encoding: %s\n","messagePattern":"\n###WARNING: Encountered abnormal bytes while converting input stream to target encoding: (.+?)\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"deps/icu-small/source/tools/toolutil/ucbuf.cpp","lineNumber":238,"sourceCode":"        source = cbuf;\n        sourceLimit = source + inputRead;\n        ucnv_toUnicode(buf->conv,&target,target+(buf->bufCapacity-offset),\n                        &source,sourceLimit,nullptr,\n                        static_cast<UBool>(buf->remaining == 0), error);\n\n        if(U_FAILURE(*error)){\n            char context[CONTEXT_LEN+1];\n            char preContext[CONTEXT_LEN+1];\n            char postContext[CONTEXT_LEN+1];\n            int8_t len = CONTEXT_LEN;\n            int32_t start=0;\n            int32_t stop =0;\n            int32_t pos =0;\n            /* use erro1 to preserve the error code */\n            UErrorCode error1 =U_ZERO_ERROR;\n            \n            if( buf->showWarning==true){\n                fprintf(stderr,\"\\n###WARNING: Encountered abnormal bytes while\"\n                               \" converting input stream to target encoding: %s\\n\",\n                               u_errorName(*error));\n            }\n\n\n            /* now get the context chars */\n            ucnv_getInvalidChars(buf->conv,context,&len,&error1);\n            context[len]= 0 ; /* null terminate the buffer */\n\n            pos = static_cast<int32_t>(source - cbuf - len);\n\n            /* for pre-context */\n            start = (pos <=CONTEXT_LEN)? 0 : (pos - (CONTEXT_LEN-1));\n            stop  = pos-len;\n\n            memcpy(preContext,cbuf+start,stop-start);\n            /* null terminate the buffer */\n            preContext[stop-start] = 0;","sourceCodeStart":220,"sourceCodeEnd":256,"githubUrl":"https://github.com/nodejs/node/blob/1b2de5e052fc0fb95fd7fb6846dcec4ade598e9e/deps/icu-small/source/tools/toolutil/ucbuf.cpp#L220-L256","documentation":"Emitted from ucbuf_fillucbuf right after ucnv_toUnicode fails. The converter was configured with UCNV_TO_U_CALLBACK_STOP, so any byte sequence illegal for the target encoding halts conversion and sets a failure status. When showWarning is true this stderr line reports the failure (via u_errorName) together with pre/post context around the offending bytes, then the code resets the converter and switches to a substitution callback so reading can continue lossily.","triggerScenarios":"Calling ucbuf_open then ucbuf_fillucbuf (directly or via the ICU tools) on a file whose bytes do not match the detected-BOM encoding or the explicitly passed *cp codepage: a file opened as UTF-8 that contains a stray Latin-1/0x80-0xFF byte, a UTF-16 file read as UTF-8, truncated multibyte sequences, or any byte illegal under ucnv_toUnicode with the STOP callback.","commonSituations":"Resource/data .txt files saved by an editor with a different encoding than declared; files transferred through tools that re-encode or strip bytes; a wrong BOM; concatenation of mixed-encoding fragments; locale data regenerated on a machine whose default encoding differs from the data's.","solutions":["Re-save the input file in the encoding ICU detected or was told to use; verify with `file -i` or `uchardet` and confirm the byte at the reported Pre/Post-context offset.","Pass the correct codepage to ucbuf_open via *cp (or fix/remove a mismatched BOM) so autodetect and explicit name agree.","If lossy conversion is acceptable, keep showWarning=true and rely on the built-in substitution callback that resumes after this warning.","Sanitize/repair the file with iconv (-c to drop, //IGNORE to substitute) before feeding it to the tool."],"exampleFix":"// before: file is Latin-1 but opened with no cp, BOM missing\nUCHARBUF* b = ucbuf_open(path, &cp, /*showWarning=*/true, /*buffered=*/false, &status);\n// after: force the real encoding so ucnv_toUnicode never hits illegal bytes\nconst char* cp = \"windows-1252\";\nUCHARBUF* b = ucbuf_open(path, &cp, /*showWarning=*/true, /*buffered=*/false, &status);","handlingStrategy":"validation","validationCode":"// Validate the file's bytes against the target encoding before ucbuf_open.\n#include <unicode/ucnv.h>\n#include <unicode/uchar.h>\nbool bytes_valid_for(const char* path, const char* cp) {\n    UErrorCode err = U_ZERO_ERROR;\n    UConverter* c = ucnv_open(cp, &err);\n    if (U_FAILURE(err) || !c) return false;\n    // read whole file into `buf`, `n`\n    // (elided: fopen/fread)\n    const char* src = buf;\n    const char* lim = buf + n;\n    UChar out[256];\n    bool ok = true;\n    while (src < lim) {\n        UChar* tgt = out;\n        UErrorCode e = U_ZERO_ERROR;\n        ucnv_toUnicode(c, &tgt, out + 256, &src, lim, nullptr, true, &e);\n        if (U_FAILURE(e)) { ok = false; break; }\n    }\n    ucnv_close(c);\n    return ok;\n}","typeGuard":null,"tryCatchPattern":"// ICU uses status codes, not exceptions. Check after each fill/read:\nUErrorCode status = U_ZERO_ERROR;\nconst char* cp = \"UTF-8\";\nUCHARBUF* buf = ucbuf_open(path, &cp, /*showWarning=*/true, false, &status);\nif (U_FAILURE(status)) { /* handle open failure */ }\nucbuf_fillucbuf(buf, &status);\nif (status == U_INVALID_CHAR_FOUND || status == U_TRUNCATED_CHAR_FOUND ||\n    status == U_ILLEGAL_CHAR_FOUND) {\n    // illegal input bytes for the encoding -- the warning has already been\n    // printed; decide whether to accept lossy substitution or abort.\n}","preventionTips":["Standardize all data inputs on UTF-8 with a BOM and reject files that fail `file -i` / `uchardet`.","Never leave *cp null when calling ucbuf_open on untrusted files; always pass an explicit canonical encoding.","Run builds with showWarning=true in CI so encoding issues surface instead of silently substituting.","Canonicalize inputs through iconv to UTF-8 before feeding them to ICU toolutil."],"tags":["icu","encoding","unicode-converter","toolutil","ucnv"],"backgroundTag":null,"analyzedSha":"1b2de5e052fc0fb95fd7fb6846dcec4ade598e9e","analyzedAt":"2026-08-13T00:53:24.642Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}