I wrote a distributed monitoring system a while back, and it has to be said that when you're testing "ping + ssh + ftp + http" on a few thousand servers nagios won't alert before your clients call.
back
1 comments
Few thousand? Impressive. FWIW, you can create a stack of Nagios monitors, where each one reports to a higher instance using NRPE or some such, instead of relegating it to a single point of failure (or in your instance, slowness).
Anything that has to ssh, ftp and http to a few thousand servers is going to be slow.
Imagine the internal monitoring that a hosting company might need. They'd have to manage:
* All the (virtual machine) host boxes.
* All the routers, switchers, and firewalls.
* Status-checks, on hosted sites, etc.
In the end I designed and implemented a system which was capable of running all the tests in around 90 seconds, by virtue of being distributed. One host does all the parsing and such like, and N-other hosts could pull out tests to execute. (As it happened we ran everything on a single box, but it was designed to be distributed, it just transpired that having 6-10 worker process pulling jobs from the queue to execute was good enough.)
Introduction:
http://blog.bytemark.co.uk/2012/12/19/custodian-a-network-mo...
Code: