Building HA Load Balancer with Nginx and keepalived

In a previous post I showed how to setup a highly available Loadbalancer using HAProxy, keepalived and Pound for SSL termination. In this post I'll demonstrate the same setup using Nginx [1].

The reason Nginx is a good load balancing solution for simpler setups is that it supports SSL termination out of the box and scales pretty well both horizontally and vertically. For what it lacks in features as compared to HAProxy, it makes up with better simplicity and extendibility through the use of modules. First let's install it:

[root@lb1 ~]# aptitude install nginx
The config file should look similar to this:
[root@lb1 ~]# cat /etc/nginx/nginx.conf 
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        
        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        include /etc/nginx/conf.d/*.conf;
        #include /etc/nginx/sites-enabled/*;
        include /etc/nginx/lb.conf;
}
The actual load balancing configuration is outsourced in the /etc/nginx/lb.conf file:
 
[root@lb1 ~]# cat /etc/nginx/lb.conf 

upstream backend  {
        ip_hash;
        server server1.example.com:8080 max_fails=2 fail_timeout=5s;
        server server2.example.com:8080 max_fails=2 fail_timeout=5s;
}

server {
        listen 192.168.1.100:80;

        rewrite ^ https://192.168.1.100/ permanent; 
}

server {
        listen 192.168.1.100:443;
        ssl on;
        ssl_certificate /etc/nginx/ssl/server.crt;
        ssl_certificate_key /etc/nginx/ssl/server.key;

        location / {
                proxy_pass http://backend;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}
Line 4 enables IP session persistence.
Line 5 and 6 specify the back-end nodes that the traffic will be routed to.
Lines 10 and 16 make Nginx listen on ports 80 and 443.
Line 12 redirects all insecure traffic arriving on port 80 to port 443.
Lines 17, 18 and 19 specify the certificate and private key files that the load balancer will use for terminating the SSL sessions.
Line 23 inserts the original client IP in the X-Forwarded-For header of the HTTP packet that the back-end nodes can use to identify where the original request came from.

To generate a self signed cert and private key for use in your test environment perform the following:

 
[root@lb1 ~]# mkdir /etc/nginx/ssl/;cd /etc/nginx/ssl/ 
[root@lb1 ~]# openssl genrsa -des3 -out server.key 1024
[root@lb1 ~]# openssl req -new -key server.key -out server.csr
[root@lb1 ~]# cp server.key server.key.org
[root@lb1 ~]# openssl rsa -in server.key.org -out server.key
[root@lb1 ~]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Line 2 generates the Private Key.
Line 3 generates a CSR (Certificate Signing Request).
Lines 4 and 5 remove the passphrase from the key.
Line 6 generates the Self-Signed Certificate

Now you are ready to start the service:

  [root@lb1 ~]# /etc/init.d/nginx start
The logs are located in /var/log/nginx.

Follow the steps in my previous post to make this a highly available service using keepalived.

Resources: [1] http://wiki.nginx.org

3 comments:

  1. i wanna the software that apply this command on :)

    ReplyDelete
  2. Is this IP:
    listen 192.168.1.100:443;

    supposed to be your keepalived VIP IP ?

    How will you make both nginx servers listen on it when you have it only on on of the lb nodes?

    ReplyDelete
  3. Yes, that's going to be the floating IP, managed by keepalived. You can start Nginx on both servers, even though the IP is raised only on one of them, by having the following kernel option enabled:
    net.ipv4.ip_nonlocal_bind=1. This will allow services to start and bind to a non raised IP.

    ReplyDelete