Sensu Monitoring for mNox

0

Sensu Setup

After a lot of research, we have realized that a good alternative to Nagios is Sensu. We did evaluate shinken, but the documentation sucks big time. Hence Sensu is a good alternative. Started off by watching some training videos on https://www.udemy.com/sensu-introduction

Once the training is thru the concepts are clear. Without the introduction the remainder of the session cannot be understood.

In my setup, I have 4 servers.

  1. Proxy – This is my http proxy
    • This server will run the sensu-server, sensu-api, redis, dashboard and rabbitmq-server.
    • This server will test reachability to the RDS, Elastic Cache and the other servers in my setup (#2, #3 and #4)
  2. Web Administration – This is my server that hosts the administration for my customer.
    • This server will serve as a sensu-client.
  3. Web Services – This is my server that hosts the webservices for my Web Administration and Mobile applications.
    • This too will serve as the sensu-client.
  4. RDS and Elastic Cache Services – This is the standard AWS services.
    • The Proxy server will test the reachability of RDS and Elastic Cache from time to time.
  5. I’ll one more external server to check the reachability of the proxy from the network. Perhaps another AWS EC2 on a different setup.
Note, we also found that sensu-client takes up about 100Mb memory
to operate. And sensu-server and sensu-api takes up about 300Mb 
to operate. So if you have a aws micro with apache, mysql, redis, 
running Sensu full stack is a bad idea.

Now, to install sensu-client on the servers, run the following commands

Install Sensu

  • Goto here to install sensu on all the servers.
  • Do not start the sensu-server and sensu-api service on the same server.
  • sudo apt-get install ruby2.0 ruby2.0-dev
  • sudo gem2.0 install sensu-plugins-disk-checks
  • sudo gem2.0 install sensu-plugins-memory-checks
  • sudo gem2.0 install sensu-plugins-network-checks
  • sudo gem2.0 install sensu-plugins-mysql-checks
  • sudo gem2.0 install sensu-plugins-cpu-checks
  • sudo gem2.0 install sensu-plugins-http-checks
  • sudo gem2.0 install sensu-plugins-process-checks
  • Once installed, copy the /etc/sensu/config.json.example to /etc/sensu/config.json

To install the needed sensu plugins

  • goto here
  • search for (control f) the needed plugins
  • Install git on your ubuntu box (sudo apt-get install git)
  • git clone the needed projects
  • Once you have git cloned the need projects
  • Copy all the files from bin folder to /etc/sensu/plugins.

Install RabbitMQ

  • https://sensuapp.org/docs/0.25/installation/install-rabbitmq-on-ubuntu-debian.html
  • Configure RabbitMQ for sensu-clients to connect
    • Change the username password on /etc/sensu/rabbitmq.json
    • This username password will be used by sensu client to connect to the rabbitmq server
    • sudo vi /etc/sensu/conf.d/rabbitmq.json
    • Remove localhost from rabbitmq and use the Internal IP address of the rabbitmq server
    • Rename user to “mcruiseon”
    • Rename password to “”
    • Retain the port to 5672
    • Open the port 5672 (only for the sensu-client IP Address)
    • At this time, errors on the sensu-client “Detected TCP connection failure” will go away
  • Configure RabbitMQ dasboard
    • Create one the rabbitmq web login password (replace username and password)
      • rabbitmqctl add_user username password
      • rabbitmqctl set_user_tags username administrator
      • rabbitmqctl set_permissions -p / username “.*” “.*” “.*”
    • On AWS
      • Open the 15672 port on the #1 server
      • Open on  your brower :15672
      • Login using the username / password you have set on rabbitmqctl (previous step)
  • Install Redis-server
    • sudo apt-get install redis-server
  • This link helped setup the dashboard

My Proxy has the following conf files for configuring the sensu server

  • api.json
{
 "api": {
 "host": "localhost",
 "port": 4567
 }
}
  • rabbitmq.json
{
 "rabbitmq": {
 "host": "0.0.0.0",
 "port": 5672,
 "vhost": "/sensu",
 "user": "yourownusername",
 "password": "yourownpassword"
 }
}
  • redis.json
{
 "redis": {
 "host": "localhost",
 "port": 6379
 }
}

My servers #2 #3 #4 are configured with sensu client. The following config file can be configured for each of my servers.

{
"rabbitmq": {
 "host": "ipaddress_of_sensu_server_Proxy_#1",
 "port": 5672,
 "user": "yourownusername",
 "password": "yourownpassword",
 "vhost": "/sensu"
 },
 "api": {
 "host": "localhost",
 "port": 4567
 },
 "checks": {
 "test": {
 "command": "echo -n OK",
 "subscribers": [
 "test"
 ],
 "interval": 60
 },
 "memory-percentage": {
 "command": "check-memory-percent.sh -w 50 -c 70",
 "interval": 10,
 "standalone": true,
 "subscribers": [
 "test"
 ]
 }
 },
 "client": {
 "name": "<name of this ubuntu box, and how you what to identify it>",
 "address": "127.0.0.1",
 "subscriptions": [
 "test"
 ]
 }
}

To check, just restart sensu-client service and run a tail on /var/log/sensu/sensu-client.log 

I ran into some trouble with sensu, stackoverflow came to help. Check out the question here.

After this I installed the dashboard on the Proxy #1 server.

  • sudo apt-get install uchiwa
  • sudo vi /etc/sensu/uchiwa.json
  • sudo update-rc.d uchiwa defaults
  • netstat -teapu | grep 3000
{
 "sensu": [
 {
 "name": "Sensu",
 "host": "127.0.0.1",
 "port": 4567,
 "timeout": 10
 },
 {
 "name": "Site 2",
 "host": "api2.example.com",
 "port": 4567,
 "ssl": false,
 "path": "",
 "user": "",
 "pass": "",
 "timeout": 10
 }
 ],
 "uchiwa": {
 "host": "0.0.0.0",
 "port": 3000,
 "refresh": 10,
 "user":"logintoyourdashboard_notsame_asrabbitmq_dontconfuseyourselfhere",
 "pass":"passwordtoyourdashboard"
 }
}

Dont forget to open up the port 3000 on AWS.

Thats it, you are done..  post questions here if you face any trouble. Also please provide feedback on this post.

Checks

Some sensu-checks need special ruby gems for installation

check-ping needs

  • sudo gem install sensu-plugins-network-checks
  • sudo sensu-install -p network-checks
  • sudo  gem install net-ping

AWS Handlers (for SES)

  • sudo gem2.0 install amazon-ses-mailer
  • sudo gem install aws-sdk