alibaba/DataX · critical · IOException

Failed to flush data to Doris. %s

Error message

Failed to flush data to Doris.
%s

What it means

DorisStreamLoadObserver.streamLoad throws this IOException when the stream load response's Status is 'Fail' — Doris explicitly rejected the load. The full response JSON (including the Status, Message, and error URL fields) is appended to the exception so the exact server-side failure reason travels with it.

Source

Thrown at doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java:72

            throw new IOException ("load_url cannot be empty, or the host cannot connect.Please check your configuration.");
        }
        String loadUrl = new StringBuilder(host)
                .append("/api/")
                .append(options.getDatabase())
                .append("/")
                .append(options.getTable())
                .append("/_stream_load")
                .toString();
        LOG.info("Start to join batch data: rows[{}] bytes[{}] label[{}].", data.getRows().size(), data.getBytes(), data.getLabel());
        Map<String, Object> loadResult = put(loadUrl, data.getLabel(), addRows(data.getRows(), data.getBytes().intValue()));
        LOG.info("StreamLoad response :{}",JSON.toJSONString(loadResult));
        final String keyStatus = "Status";
        if (null == loadResult || !loadResult.containsKey(keyStatus)) {
            throw new IOException("Unable to flush data to Doris: unknown result status.");
        }
        LOG.debug("StreamLoad response:{}",JSON.toJSONString(loadResult));
        if (RESULT_FAILED.equals(loadResult.get(keyStatus))) {
            throw new IOException(
                    new StringBuilder("Failed to flush data to Doris.\n").append(JSON.toJSONString(loadResult)).toString()
            );
        } else if (RESULT_LABEL_EXISTED.equals(loadResult.get(keyStatus))) {
            LOG.debug("StreamLoad response:{}",JSON.toJSONString(loadResult));
            checkStreamLoadState(host, data.getLabel());
        }
    }

    private void checkStreamLoadState(String host, String label) throws IOException {
        int idx = 0;
        while(true) {
            try {
                TimeUnit.SECONDS.sleep(Math.min(++idx, 5));
            } catch (InterruptedException ex) {
                break;
            }
            try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
                HttpGet httpGet = new HttpGet(new StringBuilder(host).append("/api/").append(options.getDatabase()).append("/get_load_state?label=").append(label).toString());

View on GitHub (pinned to 80ec23d5c5)

Solutions

1. Read the appended JSON in the exception — the 'Message' and 'ErrorURL' fields state the Doris-side reason; open the ErrorURL to see offending rows.
2. For quality errors, clean the data or raise max_filter_ratio in loadProps if acceptable.
3. Align the writer 'column' list (order and count) with the Doris table schema, or set the columns explicitly in loadProps.
4. If the separator appears in data, switch to a hex separator like "\\x01" or to JSON format.

Example fix

// before: CSV data contains the ',' separator
"loadProps": { "format": "csv", "column_separator": "," }
// after
"loadProps": { "format": "csv", "column_separator": "\\x01", "max_filter_ratio": "0.01" }

// reduce row rejects before they reach Doris: choose a delimiter absent from data
String sample = fetchSampleRows();
for (String cand : List.of("\\x01", "\\x1f", "\\x02")) if (!sample.contains(cand)) { chosenSep = cand; break; }

catch (IOException e) when message starts with 'Failed to flush data to Doris.': parse the appended JSON, read Status/Message/ErrorURL, fetch ErrorURL for offending rows, route them to a side output and re-flush the remainder.

When it happens

Trigger: Any Doris-side load failure: quality errors exceeding max_filter_ratio (bad rows), schema mismatch (wrong column count/order versus the table), unsupported types, invalid CSV due to a separator present in the data, or table not found. The response body in the message pinpoints which.

Common situations: Data containing the column separator itself, column list in the writer config not matching the Doris table order, NULL handling in non-nullable columns, or exceeding the filter ratio with dirty source rows.

Related errors

• Failed to create row serializer, unsupported `format` from s
• load_url cannot be empty, or the host cannot connect.Please
• Unable to flush data to Doris: unknown result status.
• Failed to flush data to Doris, Error could not get the final
• Failed to parse delimiter: `Hex str is empty`

AI-assisted analysis of alibaba/DataX@80ec23d5c5 (2026-08-14). Data as JSON: /api/errors/afedc702173fa13a. Report an issue: GitHub.