cilium/cilium · error

node error: %s

Error message

node error: %s

What it means

Relay publishes a NODE_ERROR NodeStatus when at least one Hubble node reports an error on the GetFlows request — classically an invalid flow filter that the node rejected. The message text (r.NodeStatus.Message) comes from the node and is surfaced verbatim.

Source

Thrown at cilium-cli/connectivity/check/action.go:1007

			// Handle NodeStatus messages generated by Hubble peers, containing
			// individual node readiness, unavailability, invalid filters etc.

			switch r.NodeStatus.StateChange {
			case relay.NodeState_NODE_CONNECTED:
				// Received first connection event from a Hubble peer, tentatively
				// notify the caller that traffic can be generated.
				a.Debugf("Connected to Hubble node(s) %s", r.NodeStatus.NodeNames)
				once.Do(func() { ready <- true })

			case relay.NodeState_NODE_UNAVAILABLE:
				// An unavailable node will result in the event log being incomplete,
				// so the test needs to be aborted.
				return fmt.Errorf("unavailable node(s) %s, flow results will be incomplete", r.NodeStatus.NodeNames)

			case relay.NodeState_NODE_ERROR:
				// When an invalid filter is specified, a node error will be published
				// by at least one Hubble node.
				return fmt.Errorf("node error: %s", r.NodeStatus.Message)
			}

		case *observer.GetFlowsResponse_Flow:
			// Store any flows we receive in the Action to be sent off
			// to the flow matcher later.

			a.flowsMu.Lock()
			a.flows = append(a.flows, r)
			a.flowsMu.Unlock()

		default:
			// Abort on any unknown message types.
			return fmt.Errorf("received unknown message: %q", r)

		}
	}
}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Read the node message in the error — it names the rejected filter
2. If using custom filters, validate flow filter syntax (flow filters in hubble docs) before running
3. Upgrade cilium-cli and Cilium together so filter capabilities match
4. Re-run with default filters (--flow-validation default whitelist) to isolate the custom filter

Example fix

// before
FlowFilters: []flow.Filter{{Expr: "verdict !="}} // malformed
// after
FlowFilters: []flow.Filter{{Expr: "verdict DROPPED"}}

// validate custom flow filters against a live node before the suite
_, err := hubbleClient.GetFlows(ctx, &observer.GetFlowsRequest{Whitelist: myFilters, Number: 1})
if err != nil {
    return fmt.Errorf("invalid flow filter: %w", err)
}

if err := runSuite(ctx); err != nil {
    if strings.Contains(err.Error(), "node error: ") {
        // read the node's message; it names the rejected filter
    }
    return err
}

When it happens

Trigger: A GetFlowsResponse_NodeStatus with State == relay.NodeState_NODE_ERROR arrives; typically triggered by a malformed or unsupported whitelist flow filter built by the test scenario (e.g. bad trial/requirement filters).

Common situations: Custom flow filters supplied via test parameters that the Hubble node's filter parser rejects; version mismatch where newer filter fields are sent to older Hubble agents; typo'd flow labels in custom scenarios.

Related errors

• hubble server status failure: %w
• initiating follow request: %w
• gRPC error: %w
• unavailable node(s) %s, flow results will be incomplete
• unable to create hubble client: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/959b79127aaf9448. Report an issue: GitHub.