You're already monitoring, graphing, all errors / warnings / events on your infrastructure, but do you track what happens in your clients' web browser ? Websites are now heavily relying on client-side functions, so not keeping an eye on proper JavaScript execution is terribly stupid.

JSLogger is roughly composed of :

• a JavaScript payload (payload.js) to be included in your pages which call your receiver on window.onerror() passing error details,
• a PHP script (jslog.php) which will receive and dispatch events to StatsD / Graphite & MySQL database.

Code is available on my github.
Just pull everything, create your table, edit the settings in jslog.php and URL in payload.js, include it into your pages and you're done !

Context

A few days ago, after repeated downtimes, Steve Souders twitted :

How does a small hosted site (like mine) get redundancy? Two hosting companies & DNS round robin? Any cookbook solutions?

— souders (@souders) September 21, 2012

I found this question very interesting, and here is an answer. My criteria was to build something without refactoring all my current setup, roughly composed of :

• CloudFlare as CDN in front of www.tuxz.net,
• Nagios as monitoring system,
• A blog powered by Nanoblogger,
• Dokuwiki,
• A lot of custom PHP scripts ...

All of this running in an OpenVZ container on a single physical server somewhere on the planet.
Oh ! and also a (no longer) unused free account on OpenShift (RedHat Platform as a Service) :-)

Creating origin failover site

First create a new application on OpenShift, called "failover". This application will be accessible through (depending on what your namespace is set to) : http://failover-tuxz.rhcloud.com/

Right now, your application is empty and only accessible using its default domain name. As we want it (at the end) to answer requests targeted to our main domain name, we need to add it as an alias. This operation can only be performed using the OpenShift client. The installation is quite straightforward :

$ sudo gem install rhc
$ rhc setup

You can now add your alias and use git to clone your brand new OpenShift application on your current origin, ie :

$ rhc app add-alias -a failover --alias www.tuxz.net
$ git clone ssh://[email protected]/~/git/failover.git/ /var/www/www-failover.tuxz.net/
$ tree -L 1 /var/www/
/var/www
|-- www-failover.tuxz.net
|-- www.tuxz.net

Now you just need to create a custom crontab to rsync, statify, torture then commit & push changes to your OpenShift application :

#!/bin/bash
PRIMARY_ROOT="/var/www/www.tuxz.net"
FAILOVER_ROOT="/var/www/www-failover.tuxz.net/php"
TS=`date`
rsync -rvl --delete ${PRIMARY_ROOT}/ ${FAILOVER_ROOT}/

#- do all your custom stuff here -#
cd ${FAILOVER_ROOT}/
git add .
git commit -m "www.tuxz.net - ${TS}"
git push

At this time, your OpenShift app should contain the exact (or tortured) copy of your primary origin.

Modifying DNS configuration

To make identification easier, update your DNS configuration to add 2 CNAME "www-primary" and "www-failover" pointing respectively to your primary server & your OpenShift application, then CNAME your "www" entry to "www-primary" & enable CloudFlare servies on it.
You should end up with results similar as :

$ dig -t CNAME +short www-primary.tuxz.net
fw0.tuxz.net.

$ dig -t CNAME +short www-failover.tuxz.net
failover-tuxz.rhcloud.com.

$ dig -t CNAME +short www.tuxz.net
cf-protected-www.tuxz.net.

Configuring Nagios to switch traffic to failover site in case of primary origin failure

We are going to use Nagios events handler built-in mechanism, which allow us to run scripts "when something happens".
In our case we're going to run a script interacting with CloudFlare DNS API and change the value of our origin server for our main domain.

Here is the relevant part of the Nagios configuration :

define service {
  use generic-service
  host_name www-primary.tuxz.net
  service_description Ensure that primary origin is healthy
  check_command your_command
  contact_groups admins
  max_check_attempts 4
  event_handler switch_to_failover_site
}

# commands.cfg
define command {
  command_name switch_to_failover_site
  command_line /usr/local/nagios/libexec/eventhandlers/switch_to_failover_site.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $HOSTADDRESS$ $HOSTDOWNTIME$ $SERVICEDOWNTIME$
}

And the content of switch_to_failover_site.sh :

#!/bin/sh

CLOUDFLARE_API_KEY="111111111"
CLOUDFLARE_LOGIN="[email protected]"
DNS_ZONE="example.com"
DNS_ENTRY="www.example.com"
DNS_ENTRY_ID="1111111"
DNS_ENTRY_TYPE="CNAME"
DNS_ENTRY_FAILOVER="www-failover.example.com"

__switch_to_failover() {
  /usr/bin/curl https://www.cloudflare.com/api_json.html \
    -d "a=rec_edit" \
    -d "tkn=${CLOUDFLARE_API_KEY}" \
    -d "id=${DNS_ENTRY_ID}" \
    -d "email=${CLOUDFLARE_LOGIN}" \
    -d "z=${DNS_ZONE}" \
    -d "type=${DNS_ENTRY_TYPE}" \
    -d "name=${DNS_ENTRY}" \
    -d "content=${DNS_ENTRY_FAILOVER}" \
    -d "ttl=1" \
    -d "service_mode=1"
}


[ "$1" = "CRITICAL" ] || exit 0
if [ "$2" = "SOFT" ];
then
  if [ $3 -eq 3 ];
  then
    servicestatus="$5""$6";
    [ "$servicestatus" = "00" ] && __switch_to_failover;
  fi;
fi;

Notes about this script :

• Customize variables at the top with your domain entries and cloudflare credentials / API key
• DNS_ENTRY_ID can be obtained by querying the API with the "rec_load_all" parameter (see CloudFlare API Doc)
• The script will trigger origin switch after 3 fail checks

One interesting side effect of using CloudFlare is that there's almost no DNS propagation delay. In fact your main entry is "publicly" not modified (still CNAME'd to cf-protected-www.tuxz.net) but the update is quickly propagated to the CloudFlare infrastructure. My tests shown an appox. 1 minute delay.

When having a lot of servers, relying on built-in package management tools to deploy software is a must just for :

• Dependencies
• Versioning capabilities
• File checksums / integrity
• Upgrades management
• Distribution (apt-get, yum ...)

Unfortunately, required versions are often not available, your home-made software neither. But building your own packages, no matter how good you get at it, can be a difficult and is a time consuming process.
This is where FPM will help you : you'll be able to create in a few seconds a deb, rpm, solaris package and even a puppet module from a gem, python module, folder, or npm.

FPM installation

FPM can be easily installed through gem :

$ sudo gem install fpm

The code is also available on github.

Example with nodejs

$ wget -q http://nodejs.org/dist/v0.8.8/node-v0.8.8.tar.gz
$ tar -xzf node-v0.8.8.tar.gz
$ cd node-v0.8.8/
$ ./configure --prefix=/usr
$ make
[...]
$ mkdir /tmp/node-install
$ make install DESTDIR=/tmp/node-install
[...]
$ fpm -s dir -t rpm -n node -v 0.8.8 -C /tmp/node-install usr/bin usr/lib

Et voila ! You now have a shiny rpm package that you can deploy on your own infrastructure.
Full documentation and use cases are available on the official FPM wiki.