In this post I'll setup HAProxy with SSL offloading and load balance HTTP, MySQL and rabbitmq in an active/passive mode.
First lets install HAProxy from source (you can of course use a package, but I'll be using Debian Squeeze for this deployment):
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] apt-get install libpcre3-dev libssl-dev | |
| [root@server ~] wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.4.tar.gz | |
| [root@server ~] tar zxfv haproxy-1.5.4.tar.gz | |
| [root@server ~] cd haproxy-1.5.4/ | |
| [root@server haproxy-1.5.4] make TARGET=custom CPU=native USE_PCRE=1 USE_LIBCRYPT=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_OPENSSL=yes | |
| [root@server haproxy-1.5.4] make install | |
| [root@server haproxy-1.5.4] useradd -M haproxy |
Since we are installing from source, we'll need an init script in /etc/init.d and the default options file in /etc/default. You can get one from an older package and make sure that the paths match the installation or just use the one bellow:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] cat /etc/init.d/haproxy | |
| #!/bin/sh | |
| ### BEGIN INIT INFO | |
| # Provides: haproxy | |
| # Required-Start: $local_fs $network $remote_fs | |
| # Required-Stop: $local_fs $remote_fs | |
| # Default-Start: 2 3 4 5 | |
| # Default-Stop: 0 1 6 | |
| # Short-Description: fast and reliable load balancing reverse proxy | |
| # Description: This file should be used to start and stop haproxy. | |
| ### END INIT INFO | |
| # Author: Arnaud Cornet <acornet@debian.org> | |
| PATH=/sbin:/usr/local/sbin:/bin:/usr/bin | |
| PIDFILE=/var/run/haproxy.pid | |
| CONFIG=/etc/haproxy/haproxy.cfg | |
| HAPROXY=/usr/local/sbin/haproxy | |
| EXTRAOPTS= | |
| ENABLED=0 | |
| test -x $HAPROXY || exit 0 | |
| test -f "$CONFIG" || exit 0 | |
| if [ -e /etc/default/haproxy ]; then | |
| . /etc/default/haproxy | |
| fi | |
| test "$ENABLED" != "0" || exit 0 | |
| [ -f /etc/default/rcS ] && . /etc/default/rcS | |
| . /lib/lsb/init-functions | |
| haproxy_start() | |
| { | |
| start-stop-daemon --start --pidfile "$PIDFILE" \ | |
| --exec $HAPROXY -- -f "$CONFIG" -D -p "$PIDFILE" \ | |
| $EXTRAOPTS || return 2 | |
| return 0 | |
| } | |
| haproxy_stop() | |
| { | |
| if [ ! -f $PIDFILE ] ; then | |
| # This is a success according to LSB | |
| return 0 | |
| fi | |
| for pid in $(cat $PIDFILE) ; do | |
| /bin/kill $pid || return 4 | |
| done | |
| rm -f $PIDFILE | |
| return 0 | |
| } | |
| haproxy_reload() | |
| { | |
| $HAPROXY -f "$CONFIG" -p $PIDFILE -D $EXTRAOPTS -sf $(cat $PIDFILE) \ | |
| || return 2 | |
| return 0 | |
| } | |
| haproxy_status() | |
| { | |
| if [ ! -f $PIDFILE ] ; then | |
| # program not running | |
| return 3 | |
| fi | |
| for pid in $(cat $PIDFILE) ; do | |
| if ! ps --no-headers p "$pid" | grep haproxy > /dev/null ; then | |
| # program running, bogus pidfile | |
| return 1 | |
| fi | |
| done | |
| return 0 | |
| } | |
| case "$1" in | |
| start) | |
| log_daemon_msg "Starting haproxy" "haproxy" | |
| haproxy_start | |
| ret=$? | |
| case "$ret" in | |
| 0) | |
| log_end_msg 0 | |
| ;; | |
| 1) | |
| log_end_msg 1 | |
| echo "pid file '$PIDFILE' found, haproxy not started." | |
| ;; | |
| 2) | |
| log_end_msg 1 | |
| ;; | |
| esac | |
| exit $ret | |
| ;; | |
| stop) | |
| log_daemon_msg "Stopping haproxy" "haproxy" | |
| haproxy_stop | |
| ret=$? | |
| case "$ret" in | |
| 0|1) | |
| log_end_msg 0 | |
| ;; | |
| 2) | |
| log_end_msg 1 | |
| ;; | |
| esac | |
| exit $ret | |
| ;; | |
| reload|force-reload) | |
| log_daemon_msg "Reloading haproxy" "haproxy" | |
| haproxy_reload | |
| case "$?" in | |
| 0|1) | |
| log_end_msg 0 | |
| ;; | |
| 2) | |
| log_end_msg 1 | |
| ;; | |
| esac | |
| ;; | |
| restart) | |
| log_daemon_msg "Restarting haproxy" "haproxy" | |
| haproxy_stop | |
| haproxy_start | |
| case "$?" in | |
| 0) | |
| log_end_msg 0 | |
| ;; | |
| 1) | |
| log_end_msg 1 | |
| ;; | |
| 2) | |
| log_end_msg 1 | |
| ;; | |
| esac | |
| ;; | |
| status) | |
| haproxy_status | |
| ret=$? | |
| case "$ret" in | |
| 0) | |
| echo "haproxy is running." | |
| ;; | |
| 1) | |
| echo "haproxy dead, but $PIDFILE exists." | |
| ;; | |
| *) | |
| echo "haproxy not running." | |
| ;; | |
| esac | |
| exit $ret | |
| ;; | |
| *) | |
| echo "Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}" | |
| exit 2 | |
| ;; | |
| esac | |
| : | |
| [root@server ~] chmod u+x /etc/init.d/haproxy | |
| [root@server ~] cat /etc/default/haproxy | |
| # Set ENABLED to 1 if you want the init script to start haproxy. | |
| ENABLED=1 |
Time to configure HAProxy for 4 different services - plain HTTP, HTTPS terminated by the load balancer, and MySQL and RabbitMQ in active/passive mode:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] mkdir /etc/haproxy | |
| [root@server ~] cat /etc/haproxy/haproxy.cfg | |
| global | |
| log 127.0.0.1 local1 | |
| log-tag haproxy | |
| maxconn 4096 | |
| user haproxy | |
| group haproxy | |
| daemon | |
| stats socket /var/run/haproxy.sock mode 600 level admin | |
| stats timeout 2m | |
| tune.ssl.default-dh-param 2048 | |
| defaults | |
| log global | |
| mode http | |
| timeout connect 5000ms | |
| timeout client 50000ms | |
| timeout server 50000ms | |
| option dontlognull | |
| option http-server-close | |
| peers HAPEERS | |
| peer haproxy-n01 10.13.238.51:5555 | |
| peer haproxy-n02 10.13.238.51:5555 | |
| frontend http | |
| bind 10.13.238.21:80 | |
| reqadd X-Forwarded-Proto:\ http | |
| default_backend http_nodes | |
| frontend https-terminated | |
| bind 10.13.238.22:443 ssl crt /etc/ssl/certs/api.example.net.pem ciphers RC4+SHA:!aNULL:!MD5 no-sslv3 | |
| option httpclose | |
| option forwardfor | |
| option httplog clf | |
| reqadd X-Forwarded-Proto:\ https | |
| default_backend https-terminated_nodes | |
| frontend mysql | |
| mode tcp | |
| bind 10.13.238.19:3306 | |
| timeout client 168h | |
| default_backend mysql_nodes | |
| log global | |
| option tcplog | |
| option logasap | |
| frontend rabbitmq | |
| mode tcp | |
| bind 10.13.238.20:5672 | |
| timeout client 168h | |
| default_backend rabbitmq_nodes | |
| log global | |
| option tcplog | |
| option logasap | |
| backend http_nodes | |
| mode http | |
| balance leastconn | |
| option httpclose | |
| option forwardfor | |
| option redispatch | |
| # This check will fail a node on codes that are not 2xx or 3xx | |
| option httpchk GET / | |
| # This check will fail a node on 5xx codes only | |
| # http-check expect ! rstatus ^5 | |
| cookie JSESSIONID prefix | |
| server apt-n01 10.13.238.39:80 check inter 5000 | |
| backend https-terminated_nodes | |
| mode http | |
| balance roundrobin | |
| cookie JSESSIONID prefix | |
| option redispatch | |
| option httpchk GET / | |
| server api-n01 10.13.238.37:8888 check inter 5000 | |
| server api-n02 10.13.238.38:8888 check inter 5000 | |
| backend mysql_nodes | |
| mode tcp | |
| balance roundrobin | |
| option mysql-check user haproxy_check | |
| stick-table type ip size 20k peers HAPEERS | |
| stick on dst | |
| timeout server 168h | |
| server mysql-n01 10.13.238.53:3306 check | |
| server mysql-n02 10.13.238.54:3306 check backup | |
| backend rabbitmq_nodes | |
| mode tcp | |
| balance roundrobin | |
| stick-table type ip size 20k peers HAPEERS | |
| stick on dst | |
| timeout server 168h | |
| server rabbitmq-n01 10.13.238.35:5672 check | |
| server rabbitmq-n02 10.13.238.36:5672 check backup | |
All of the options are documented [1], but the most interesting once are the "stats socket", which allows you to connect to a socket and perform simple operations on HAProxy and gather statistics without needing to restart the server. You can do this in either interactive or non-interactive mode using the socat utility:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] apt-get install socat | |
| [root@server ~] socat /var/run/haproxy.sock readline | |
| # Or on newer versions of socat: | |
| # socat /var/run/haproxy.sock stdio | |
| prompt | |
| > show info | |
| Name: HAProxy | |
| Version: 1.5.4 | |
| Release_date: 2014/09/02 | |
| Nbproc: 1 | |
| Process_num: 1 | |
| Pid: 17066 | |
| Uptime: 0d 1h07m55s | |
| Uptime_sec: 4075 | |
| Memmax_MB: 0 | |
| Ulimit-n: 8248 | |
| Maxsock: 8248 | |
| Maxconn: 4096 | |
| Hard_maxconn: 4096 | |
| CurrConns: 313 | |
| CumConns: 1792 | |
| CumReq: 27874 | |
| MaxSslConns: 0 | |
| CurrSslConns: 0 | |
| CumSslConns: 537 | |
| Maxpipes: 0 | |
| PipesUsed: 0 | |
| PipesFree: 0 | |
| ConnRate: 1 | |
| ConnRateLimit: 0 | |
| MaxConnRate: 68 | |
| SessRate: 1 | |
| SessRateLimit: 0 | |
| MaxSessRate: 68 | |
| SslRate: 0 | |
| SslRateLimit: 0 | |
| MaxSslRate: 5 | |
| SslFrontendKeyRate: 0 | |
| SslFrontendMaxKeyRate: 6 | |
| SslFrontendSessionReuse_pct: 0 | |
| SslBackendKeyRate: 0 | |
| SslBackendMaxKeyRate: 0 | |
| SslCacheLookups: 0 | |
| SslCacheMisses: 0 | |
| CompressBpsIn: 0 | |
| CompressBpsOut: 0 | |
| CompressBpsRateLim: 0 | |
| Tasks: 345 | |
| Run_queue: 1 | |
| Idle_pct: 100 | |
| node: zxtm-n01 | |
| description: | |
| > quit | |
| [root@server ~] echo "show stat" | socat - /var/run/haproxy.sock | grep -E "^rabbit|pxname" | tr -s ',' ' ' | tr -d '#' | column -t | |
| pxname svname qcur qmax scur smax slim stot bin bout dreq dresp ereq econ eresp wretr wredis status weight act bck chkfail chkdown lastchg downtime qlimit pid iid sid throttle lbtot tracked type rate rate_lim rate_max check_status check_code check_duration hrsp_1xx hrsp_2xx hrsp_3xx hrsp_4xx hrsp_5xx hrsp_other hanafail req_rate req_rate_max req_tot cli_abrt srv_abrt comp_in comp_out comp_byp comp_rsp lastsess last_chk last_agt qtime ctime rtime ttime | |
| rabbitmq FRONTEND 194 194 2000 194 0 0 0 0 0 OPEN 1 6 0 0 0 0 44 0 0 0 0 0 0 0 | |
| rabbitmq_nodes rabbitmq-n01 0 0 194 194 194 0 0 0 0 0 0 0 UP 1 1 0 0 0 740 0 1 14 1 1 2 0 44 L4OK 0 0 0 0 592 0 0 0 0 | |
| rabbitmq_nodes rabbitmq-n02 0 0 0 0 0 0 0 0 0 0 0 0 UP 1 0 1 0 0 740 0 1 14 2 0 2 0 0 L4OK 0 0 0 0 -1 0 0 0 0 | |
| rabbitmq_nodes BACKEND 0 0 194 194 2000 194 0 0 0 0 0 0 0 0 UP 1 1 1 0 740 0 1 14 0 1 1 0 44 0 0 0 0 0 0 592 0 0 0 0 | |
| [root@server ~] echo "disable server http_nodes/apt-n01" | socat stdio /var/run/haproxy.sock |
To enable the static HTTP status page add the following to the config file:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| listen stats :8080 | |
| mode http | |
| stats enable | |
| stats hide-version | |
| stats realm Haproxy\ Statistics | |
| stats uri / | |
| stats auth admin:yourpassword |
To have some real time view of what's going on, you can use the ncurses hatop, just install and run:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] apt-get install hatop | |
| [root@server ~] hatop -s /var/run/haproxy.sock |
The SSL termination is configured by specifying the "ssl" option and providing a PEM file, containing the cert and the private key on line 34.
To setup MySQL and RabbitMQ in an active/passive mode, where only one of the servers will be accepting connections, and when the main server comes back up after a failure the connections will still stick to the backup server, you specify the "backup" option and define a stick-table on line 87 and line 83.
One more option worth mentioning is the MySQL health check specified on line 82. The check consists of sending two MySQL packets, one Client Authentication packet, and one QUIT packet, to correctly close the MySQL session, using the specified user - in this case haproxy_check - so make sure you create it first:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| mysql> USE mysql; | |
| mysql> INSERT INTO user (Host,User) values ('10.13.238.51','haproxy_check'); | |
| mysql> FLUSH PRIVILEGES; |
Another useful option is the tcp-check. With this it's easy to send and receive certain messages to the back-ends and act on them. This is particularly useful in managing Redis master and slave instances controlled by Sentinel:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| backend redis_nodes | |
| mode tcp | |
| option tcplog | |
| option tcp-check | |
| tcp-check send AUTH\ somepassword\r\n | |
| tcp-check expect string +OK | |
| tcp-check send PING\r\n | |
| tcp-check expect string +PONG | |
| tcp-check send info\ replication\r\n | |
| tcp-check expect string role:master | |
| tcp-check send QUIT\r\n | |
| tcp-check expect string +OK | |
| server redis_node1 192.168.1.10:6379 maxconn 4096 check inter 5s | |
| server redis_node2 192.168.1.20:6379 maxconn 4096 check inter 5s | |
| server redis_node3 192.168.1.30:6379 maxconn 4096 check inter 5s |
To allow HAProxy to send separate logs to rsyslog you can use a file similar to this:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] cat /etc/rsyslog.d/haproxy.conf | |
| $ModLoad imudp | |
| $UDPServerAddress 127.0.0.1 | |
| $UDPServerRun 514 | |
| if $syslogfacility-text == 'local1' and $msg contains 'rabbitmq' then /var/log/haproxy_rabbitmq.log | |
| if $syslogfacility-text == 'local1' and $msg contains 'http' then /var/log/haproxy_api.log | |
| if $syslogfacility-text == 'local1' and $msg contains 'mysql' then /var/log/haproxy_mysql.log | |
| if $syslogfacility-text == 'local1' and ($msg contains 'UP' or $msg contains 'DOWN') then /var/log/haproxy_backends.log | |
| local1.* -/var/log/haproxy.log | |
| & ~ |
And finally start the service:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@server ~] /etc/init.d/rsyslog restart | |
| [root@server ~] /etc/init.d/haproxy start |
Resources:
[1]. http://cbonte.github.io/haproxy-dconv/configuration-1.5.html
