If you receive the following message:
Cluster host connection failure for Local server: Connection refused (111)
...then the cc-control process has exited for some reason.
The very first steps you should take when diagnosing this problem are as follows:
- Ensure that cc-control hasn't simply been shut down. Try running:
/etc/init.d/centovacast start
- If cc-control starts up and the problem does not recur, then there is nothing to diagnose and you can stop here.
- If cc-control gives an error at this point, then send us the error. No need to troubleshoot beyond this point.
- Normally, Centova Cast's cron job will automatically restart cc-control within 60 seconds if it exits for ANY reason, so if it remains down for more than a minute, it's likely that there is a problem with your cron job or that your firewall is blocking access to localhost on port 2198. Check your cron logs (/var/log/cron or /var/log/messages) to determine why the "/etc/init.d/centovacast check" cron job is not running correctly. This cron job should be configured in /etc/cron.d/centovacast.
- If you believe cc-control is crashing and wish to troubleshoot cc-control to determine why, continue reading, but please read carefully. As explained below, after enabling core dumps, you MUST wait until the next time cc-control crashes.
Debugging a crash
If cc-control is actually crashing and you wish to determine why, you can diagnose it as follows.
Update Centova Cast to the latest build
If you believe cc-control is in fact crashing with a segfault, run the update command to ensure that you are running the very latest build.
The data collected by the procedure below relies on values built into the executable file which change every time we rebuild cc-control. Accordingly, the data is only useful if we test it against the exact same build of cc-control that you are using. If you are using an outdated build of cc-control -- even if it's just a couple of days old -- the data will be totally useless to us.
So even if you think that you are running the latest build, update anyway:
/usr/local/centovacast/sbin/update
Enable core dumps
Enable core dumps on your server by running the following command as root:
/usr/local/centovacast/sbin/enable_coredumps start
Wait for a core dump
Periodically check /var/spool/coredumps and look for files named core.cc-control_*. As soon as at least one such file exists, zip (or tar/gzip) it up and send it to us in a support ticket.
Disable core dumps
Once you've sent us a core dump, disable core dumps by running:
/usr/local/centovacast/sbin/enable_coredumps stop
Also note that recent builds of Centova Cast include crash detection and recovery code. So in the event of a control daemon crash, Centova Cast will automatically restart cc-control within a minute or two, and you may not even notice it was down. So be sure to periodically check the debug log file to see if a crash has been recorded.
