• a JavaScript payload (payload.js) to be included in your pages which call your receiver on window.onerror() passing error details,
• a PHP script (jslog.php) which will receive and dispatch events to StatsD / Graphite & MySQL database.

Code is available on my github.
Just pull everything, create your table, edit the settings in jslog.php and URL in payload.js, include it into your pages and you're done !

How does a small hosted site (like mine) get redundancy? Two hosting companies & DNS round robin? Any cookbook solutions?

— souders (@souders) September 21, 2012

I found this question very interesting, and here is an answer. My criteria was to build something without refactoring all my current setup, roughly composed of :

• CloudFlare as CDN in front of www.tuxz.net,
• Nagios as monitoring system,
• A blog powered by Nanoblogger,
• Dokuwiki,
• A lot of custom PHP scripts ...

Creating origin failover site

First create a new application on OpenShift, called "failover". This application will be accessible through (depending on what your namespace is set to) : http://failover-tuxz.rhcloud.com/

Right now, your application is empty and only accessible using its default domain name. As we want it (at the end) to answer requests targeted to our main domain name, we need to add it as an alias. This operation can only be performed using the OpenShift client. The installation is quite straightforward :

$ sudo gem install rhc
$ rhc setup

You can now add your alias and use git to clone your brand new OpenShift application on your current origin, ie :

$ rhc app add-alias -a failover --alias www.tuxz.net
$ git clone ssh://xxxxxxxxxxxxx@failover-tuxz.rhcloud.com/~/git/failover.git/ /var/www/www-failover.tuxz.net/
$ tree -L 1 /var/www/
/var/www
|-- www-failover.tuxz.net
|-- www.tuxz.net

Now you just need to create a custom crontab to rsync, statify, torture then commit & push changes to your OpenShift application :

#!/bin/bash
PRIMARY_ROOT="/var/www/www.tuxz.net"
FAILOVER_ROOT="/var/www/www-failover.tuxz.net/php"
TS=`date`
rsync -rvl --delete ${PRIMARY_ROOT}/ ${FAILOVER_ROOT}/

#- do all your custom stuff here -#
cd ${FAILOVER_ROOT}/
git add .
git commit -m "www.tuxz.net - ${TS}"
git push

At this time, your OpenShift app should contain the exact (or tortured) copy of your primary origin.

Modifying DNS configuration

To make identification easier, update your DNS configuration to add 2 CNAME "www-primary" and "www-failover" pointing respectively to your primary server & your OpenShift application, then CNAME your "www" entry to "www-primary" & enable CloudFlare servies on it.
You should end up with results similar as :

$ dig -t CNAME +short www-primary.tuxz.net
fw0.tuxz.net.

$ dig -t CNAME +short www-failover.tuxz.net
failover-tuxz.rhcloud.com.

$ dig -t CNAME +short www.tuxz.net
cf-protected-www.tuxz.net.

Configuring Nagios to switch traffic to failover site in case of primary origin failure

We are going to use Nagios events handler built-in mechanism, which allow us to run scripts "when something happens".
In our case we're going to run a script interacting with CloudFlare DNS API and change the value of our origin server for our main domain.

Here is the relevant part of the Nagios configuration :

define service {
  use generic-service
  host_name www-primary.tuxz.net
  service_description Ensure that primary origin is healthy
  check_command your_command
  contact_groups admins
  max_check_attempts 4
  event_handler switch_to_failover_site
}

# commands.cfg
define command {
  command_name switch_to_failover_site
  command_line /usr/local/nagios/libexec/eventhandlers/switch_to_failover_site.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $HOSTADDRESS$ $HOSTDOWNTIME$ $SERVICEDOWNTIME$
}

And the content of switch_to_failover_site.sh :

#!/bin/sh

CLOUDFLARE_API_KEY="111111111"
CLOUDFLARE_LOGIN="toto@example.com"
DNS_ZONE="example.com"
DNS_ENTRY="www.example.com"
DNS_ENTRY_ID="1111111"
DNS_ENTRY_TYPE="CNAME"
DNS_ENTRY_FAILOVER="www-failover.example.com"

__switch_to_failover() {
  /usr/bin/curl https://www.cloudflare.com/api_json.html \
    -d "a=rec_edit" \
    -d "tkn=${CLOUDFLARE_API_KEY}" \
    -d "id=${DNS_ENTRY_ID}" \
    -d "email=${CLOUDFLARE_LOGIN}" \
    -d "z=${DNS_ZONE}" \
    -d "type=${DNS_ENTRY_TYPE}" \
    -d "name=${DNS_ENTRY}" \
    -d "content=${DNS_ENTRY_FAILOVER}" \
    -d "ttl=1" \
    -d "service_mode=1"
}


[ "$1" = "CRITICAL" ] || exit 0
if [ "$2" = "SOFT" ];
then
  if [ $3 -eq 3 ];
  then
    servicestatus="$5""$6";
    [ "$servicestatus" = "00" ] && __switch_to_failover;
  fi;
fi;

Notes about this script :

• Customize variables at the top with your domain entries and cloudflare credentials / API key
• DNS_ENTRY_ID can be obtained by querying the API with the "rec_load_all" parameter (see CloudFlare API Doc)
• The script will trigger origin switch after 3 fail checks

• Dependencies
• Versioning capabilities
• File checksums / integrity
• Upgrades management
• Distribution (apt-get, yum ...)

FPM installation

FPM can be easily installed through gem :

$ sudo gem install fpm

The code is also available on github.

Example with nodejs

$ wget -q http://nodejs.org/dist/v0.8.8/node-v0.8.8.tar.gz
$ tar -xzf node-v0.8.8.tar.gz
$ cd node-v0.8.8/
$ ./configure --prefix=/usr
$ make
[...]
$ mkdir /tmp/node-install
$ make install DESTDIR=/tmp/node-install
[...]
$ fpm -s dir -t rpm -n node -v 0.8.8 -C /tmp/node-install usr/bin usr/lib

Et voila ! You now have a shiny rpm package that you can deploy on your own infrastructure.
Full documentation and use cases are available on the official FPM wiki.

X-XSS-Protection

This header is only interpreted by IE 8 & 9.
It turns on XSS (Cross-Site-Scripting) protection which is turned off by default as it could potentially break some websites. To turn on the XSS filter, inject the header X-XSS-Protection "1; mode=block". Sending a value of "0" enforce the protection to be disabled.

X-XSS-Protection: "1; mode=block";

X-Content-Type-Options: "nosniff"

This header protects browser from "mime" based attacks. It will prevent IE from MIME-sniffing a response away from the declared Content-Type. So if the server says the content is "text/html", the browser will render it as "text/html".

X-Content-Type-Options: "nosniff";

X-Frame-Options

The X-Frame-Options HTTP response header is used to indicate whether or not a browser is allowed to render a page in a <frame> or <iframe>. This can be used to avoid clickjacking attacks, by ensuring that your website will never be embedded into a malicious website.

Three values are supported :

• DENY : Prevents any page to be rendered if loaded into a <frame> or <iframe> even from the same domain.
• SAMEORIGIN : Prevents any page to be rendered if loaded into a <frame> or <iframe> from an external website.
• ALLOW-FROM origin : Prevent any page to be rendered if if the origin of the top-level browsing context is different than the origin value supplied with the Allow-From directive.

X-Frame-Options "DENY";

X-Content-Security-Policy

This header is designed to specify how content interacts with your website. It helps mitigate and detect types of attacks such as XSS and data injection. Important: all the inline scripts are prohibited by default when using CSP.

Multiple combinations are possible :

X-Content-Security-Policy: allow 'self'

X-Content-Security-Policy: allow 'self'; img-src *; \
                           object-src *.cdn.com; \
                           script-src trustedscripts.example.com

X-Content-Security-Policy: allow https://*:443

You can also specify multiple headers (ie: server wide level + project specific directives) :

X-Content-Security-Policy: allow *; script-src 'self'
X-Content-Security-Policy: allow *; script-src 'self'; media-src 'self';

I've seen yesterday night something which could be described as a "DDOS attack" on one of the infrastructure I'm managing : during approx. 7h we've received a continuous huge amount of connections / HTTP requests coming from more than 1800 differents IP, mainly located in :

• Russian Federation
• Ukraine

User-Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620	

After digging into google, it seems that many people had the same experience with this crawler and that requesting rate-limiting was not successful. Moreover, some people also describe the fact that denying 80legs in the robots.txt was not sufficient to prevent them to crawl you. So, in this case I suggest you to put preventive rules either in your Web Application Firewall or in your load-balancer / webserver to prevent them reaching / overloading your web infrastructure.

Below an example of HAProxy configuration to tarpit all HTTP requests from this crawler :

frontend HTTP
	[...]
        #
        # Block all requests from 80legs
        #
        reqitarpit ^User-Agent:.*80legs*

One cool thing : they provide the full IP ranges of their platform so you can easily lock down your origin to only accept connections coming from the CloudFlare network.
Unfortunately, I'm hosting websites that don't use CloudFlare, so I can't put these restrictions directly in my firewall ... so let's have some fun at L7 with HAProxy & ACL feature.

The use of Access Control Lists (ACL) provides a flexible solution to perform content switching and generally to take decisions based on content extracted from the request, the response or any environmental status. You can combine them to decide what to do with an incoming request (block, pass to a backend server ...).
In my case, I want to block all requests not targetted to the hosting platform which are not coming through the CloudFlare network, see below for the corresponding HAProxy configuration section :

frontend HTTP
	[...]
	#
	# Block all requests not coming from CloudFlare Network
	#
	acl cloudflare_valid_ip src -f /etc/haproxy/cloudflareIPs
	block if !host_hdr_hosting !cloudflare_valid_ip

	use_backend web.tuxz.net if host_hdr_tuxz
	use_backend hosting.tuxz.net if host_hdr_hosting

The "/etc/haproxy/cloudflareIPs" is basically a local copy of https://www.cloudflare.com/ips-v4, which is updated each time a new IP Range is added. Don't forget to follow updates !

RewriteEngine On
RewriteRule ^([A-Z]{2})_([a-z]{2})$  /rewrite.php?a=$1&b=$2 [L,E=addheader:1]
Header set my-header "myvalue" env=REDIRECT_addheader

#!/bin/bash
CONFIG_FILE="config.ini"
SECTION="section_1"

eval `sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
    -e 's/;.*$//' \
    -e 's/[[:space:]]*$//' \
    -e 's/^[[:space:]]*//' \
    -e "s/^\(.*\)=\([^\"']*\)$/\1=\"\2\"/" \
   < $CONFIG_FILE \
    | sed -n -e "/^\[$SECTION\]/,/^\s*\[/{/^[^;].*\=.*/p;}"`

To remove confusion with the license covering my articles, this snippet is available under the WTFPL license.

            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
                    Version 2, December 2004

 Copyright (C) 2004 Sam Hocevar 

 Everyone is permitted to copy and distribute verbatim or modified
 copies of this license document, and changing it is allowed as long
 as the name is changed.

            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. You just DO WHAT THE FUCK YOU WANT TO.