Item — HN Reader

I'm intrigued. I collect real-time machine performance data using a UDP multicast daemon (written in C). It's fallible on some network topologies, etc., but it's extremely light and efficient.

I wonder if that could be plugged in to Prometheus without the overhead of HTTP collection...

2 comments

HTTP collection doesn't really add a lot of overhead.

You might be interested in the https://github.com/prometheus/collectd_exporter as collectd works in a similar way.

For machine monitoring http://www.boxever.com/monitoring-your-machines-with-prometh... covers how to set it up with Prometheus - it's pretty easy to get working.

It does in the sense that an HTTP connection (either on the server or the client side) will expend roughly 20 times the CPU and buffers than generating or handling a single UDP packet. It all adds up on small systems.

You'd usually transfer many metrics in a single HTTP request, so the cost is amortized in most cases.

I can pack a lot on a UDP packet as well...

Not sure if this is what you're comparing it to, but be aware that Prometheus' approach is fundamentally different from StatsD-like approaches where you send every event or a subsampling thereof to a monitoring server.

Prometheus is state-based, not event-based. It only stops by your monitored instances once every couple of seconds and gathers their current state. E.g. for counting events, clients simply expose cumulative counters over their lifetime which they can increment locally in memory, and Prometheus comes by for example every 15 or 30 seconds and stores the current counter state.

The HTTP traffic incurred in this case is not really a problem and you'll usually run into bottlenecks at other places (like storage sample ingestion) before you run into network transfer bottlenecks.

There's a problem with metrics over UDP: an overloaded server will not be sending its metrics, so it will appear as if it's doing ok.

Your assumption is wrong, since lack of data points in a given time interval is, in itself, a data point. If metrics don't arrive on time after a set interval, the machine is flagged.