Puppet is a tool designed to manage the configuration of systems declaratively. The user describes system resources and their state, either using Puppet's declarative language or a Ruby DSL (domain-specific language). This information is stored in files called "Puppet manifests". Puppet discovers the system information via a utility called Facter, and compiles the Puppet manifests into a system-specific catalog containing resources and resource dependency, which are applied against the target systems. Any actions taken by Puppet are then reported.
This all works great for a relatively small number of servers. After about 3000 servers, a single Puppet Master starts to slow down, as most of the Puppet operations are CPU bound. In some cases spreading the requests from the Puppet Agents randomly in time helps, but with thousands of servers this might not be a good option, as it will take too much time for a change to propagate to all clients.
There are many ways of scaling the Puppet infrastructure, because scaling Puppet is a problem of scaling HTTP. One can use hardware or software load balancers like F5 BigIP, Zeus/Stingray, HAProxy, Pound etc. For the purpose of this writing I'll use Apache with mod_passenger, mod_proxy and separate dedicated Active/Passive Certificate Authority.
The infrastructure will consist of the following servers:
* Server puppetlb.example.com, that will be the dedicated LB, running Apache with mod_proxy, mod_proxy_balancer, mod_headers and mod_ssl, which will terminate the SSL traffic, perform a certificate validation for the Puppet Agents and forward request to a cluster of CA's and Puppet Masters, depending on the requests. There will be a passive LB, with both running keepalived with a floating VIP.
* Servers puppetca1.example.com and puppetca2.example.com (active/passive) that will be responsible for signing the certificate requests from the Puppet Agents and exporting the CA to be used by the load balancer for validation.
* Servers puppetmaster1.example.com and puppetmaster2.example.com are the Puppet Masters that will host all manifests. They need to share all catalogs, by either pulling from a git repository, rsyncing between each other or by any other way of shared storage.
It is important to understand that when your master is running behind an Apache proxy the proxy is the SSL endpoint. It does all the validation and authentication of the node and traffic between the proxy and the masters happens in clear text.
The master knows the client has been authenticated because the proxy adds an HTTP header that says so (usually X-Client-Verify for Apache/Passenger).
1. If the client runs for the 1st time, it generates a Certificate Signing Request and a private key. The former is an x509 certificate that is self-signed.
2. The client connects to the master (at this time the client is not authenticated) and sends its CSR, it will also receives the CA certificate and the CRL in return.
3. The master stores locally the CSR.
4. The administrator checks the CSR and can eventually sign it (this process can be automated with autosigning).
5. The client is then waiting for his signed certificate, which the master ultimately sends.
6. All next communications will use this client certificate. Both the master and client will authenticate each others by virtue of sharing the same CA.
The infrastructure will look something like this:
Configuring the Load balancer
On puppetlb.example.com install Apache and enable the following modules:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetlb~]$ apt-get install apache2 | |
| [root@puppetlb~]$ a2enmod proxy | |
| [root@puppetlb~]$ a2enmod headers | |
| [root@puppetlb~]$ a2enmod proxy_balancer | |
| [root@puppetlb~]$ a2enmod ssl | |
| [root@puppetlb~]$ a2enmod proxy_http |
Create a virtual host file that will define the load balancer:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ vi /etc/apache2/sites-available/puppetlb.conf | |
| <Proxy balancer://puppetmaster> | |
| BalancerMember http://10.181.101.213:8140 | |
| BalancerMember http://10.176.69.55:8140 | |
| </Proxy> | |
| <Proxy balancer://puppetmasterca> | |
| BalancerMember http://10.181.101.249:8140 | |
| BalancerMember http://10.181.101.250:8140 status=+H | |
| </Proxy> | |
| Listen 8140 | |
| <VirtualHost *:8140> | |
| SSLEngine on | |
| SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA | |
| SSLProtocol -ALL +SSLv3 +TLSv1 | |
| SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP | |
| SSLCertificateFile /var/lib/puppet/ssl/certs/puppetlb.example.com.pem | |
| SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/puppetlb.example.com.pem | |
| SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem | |
| SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem | |
| SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem | |
| SSLVerifyClient optional | |
| SSLVerifyDepth 1 | |
| SSLOptions +StdEnvVars | |
| RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e | |
| RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e | |
| RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e | |
| <Location /> | |
| SetHandler balancer-manager | |
| Order allow,deny | |
| Allow from all | |
| </Location> | |
| ProxyPassMatch ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca | |
| ProxyPassReverse ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca | |
| # Direct all other Puppet agent requests to the default set of workers. | |
| ProxyPass / balancer://puppetmaster/ | |
| ProxyPassReverse / balancer://puppetmaster/ | |
| ProxyPreserveHost On | |
| </VirtualHost> | |
| [root@puppetlb~]$ a2ensite puppetlb.conf |
Create the cert directory that will be mounted via NFS from the active CA server puppetca1.example.com later on.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetlb ~]$ mkdir -p /var/lib/puppet/ssl |
Do not start Apache just yet, as it will complain about the missing SSL cert.
All this config does is define two sets of back-end workers that will be responsible for handling either the cert requests or delivering the compiled manifests from the Puppet Masters. In other words if a request from a Puppet agent comes to the LB that contains the "certificate" word in it's URL then mod_proxy will forward it to the puppetca1, for signing, etc. All other requests will be forwarded to the Puppet Masters, puppetmaster1 and puppetmaster2 in a round-robin fashion. If one of the Puppet Masters fails, mod_proxy will remove it from rotation.
The option status=+H tells the front end that the second CA member is a hot
standby and will not receive any requests until puppetca1 fails.
Configuring the Puppet CA nodes
On puppetca1 and puppetca2 install apache, mod_passenger, puppet and ruby. Since Puppet will be run by Apache Passenger make sure puppetmasterd does not start at boot, as it will use WeBrick by default, and conflict with Apache Passenger.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list | |
| [root@puppetca1~]$ apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30 | |
| [root@puppetca1~]$ apt-get update | |
| [root@puppetca1~]$ apt-get install ruby libshadow-ruby1.8 | |
| [root@puppetca1~]$ apt-get install puppetmaster puppet facter | |
| [root@puppetca1~]$ apt-get install apache2 libapache2-mod-passenger | |
| [root@puppetca1~]$ /etc/init.d/puppetmaster stop | |
| [root@puppetca1~]$ update-rc.d -f puppetmaster remove |
puppetmasterd will start by default, generating a CA with the name of the hostname - puppetca1. This is not what we want, as the cert name needs to have the load balancer name in it, because the certs will be validated on the load balancer. Make sure you stop puppetmasterd and change the puppet config file:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ vi /etc/puppet/puppet.conf | |
| [main] | |
| logdir=/var/log/puppet | |
| vardir=/var/lib/puppet | |
| ssldir=/var/lib/puppet/ssl | |
| rundir=/var/run/puppet | |
| factpath=$vardir/lib/facter | |
| templatedir=$confdir/templates | |
| certname = puppetlb.example.com | |
| [master] | |
| # These are needed when the puppetmaster is run by passenger | |
| # and can safely be removed if webrick is used. | |
| ssl_client_verify_header = HTTP_X_CLIENT_VERIFY | |
| ssl_client_header = HTTP_X_CLIENT_DN |
After the config change, run puppetmasterd to generate the new CA with the puppetlb.example.com name, that will match the hostname of the load balancer:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ puppetmasterd | |
| [root@puppetca1~]$ /etc/init.d/puppetmaster stop |
This will generate the new CA cert, private and public key in /var/lib/puppet/ssl with the name puppetlb.example.com, which is the hostname of the load balancer.
Now let's export the CA directory via NFS, so that the load balancer can mount it and use it to validate the puppet agents:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ apt-get install nfs-kernel-server portmap | |
| [root@puppetca1~]$ vi /etc/exports | |
| /var/lib/puppet/ssl *(rw,sync,no_root_squash) | |
| [root@puppetca1~]$ /etc/init.d/nfs-kernel-server start |
Time to enable the Apache modules:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ a2enmod passenger | |
| [root@puppetca1~]$ a2enmod proxy | |
| [root@puppetca1~]$ a2enmod proxy_balancer | |
| [root@puppetca1~]$ a2enmod proxy_http | |
| [root@puppetca1~]$ a2enmod headers |
Edit the passenger config file:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ vi /etc/apache2/mods-available/passenger.conf | |
| <IfModule mod_passenger.c> | |
| PassengerRoot /usr | |
| PassengerRuby /usr/bin/ruby | |
| # Recommended Passenger Configuration | |
| PassengerHighPerformance on | |
| PassengerUseGlobalQueue on | |
| # PassengerMaxPoolSize control number of application instances, | |
| # typically 1.5x the number of processor cores. | |
| PassengerMaxPoolSize 6 | |
| # Restart ruby process after handling specific number of request to resolve MRI memory leak. | |
| PassengerMaxRequests 4000 | |
| # Shutdown idle Passenger instances after 30 min. | |
| PassengerPoolIdleTime 1800 | |
| </IfModule> |
Let's define the Apache virtual host that will do all the work. Notice that the CA does not use SSL anymore, as it is terminated on the load balancer:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ vim.tiny /etc/apache2/sites-available/puppetmaster.conf | |
| Listen 8140 | |
| <VirtualHost *:8140> | |
| SSLEngine off | |
| # Obtain Authentication Information from Client Request Headers | |
| SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1 | |
| SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1 | |
| RackAutoDetect On | |
| DocumentRoot /etc/puppet/rack/puppetmaster/public/ | |
| <Directory /etc/puppet/rack/puppetmaster/> | |
| Options None | |
| AllowOverride None | |
| Order allow,deny | |
| allow from all | |
| </Directory> | |
| </VirtualHost> |
Configure the rack file that Passenger uses and restart Apache:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ mkdir -p /etc/puppet/rack/puppetmaster/{public,tmp} | |
| [root@puppetca1~]$ vi /etc/puppet/rack/puppetmaster/config.ru | |
| # a config.ru, for use with every rack-compatible webserver. | |
| $0 = "master" | |
| # if you want debugging: | |
| # ARGV << "--debug" | |
| ARGV << "--rack" | |
| require 'puppet/application/master' | |
| run Puppet::Application[:master].run | |
| # EOF /etc/puppet/rack/puppetmaster/config.ru | |
| [root@puppetca1~]$ chown puppet:puppet /etc/puppet/rack/puppetmaster/config.ru | |
| [root@puppetca1~]$ /etc/init.d/apache restart |
On the load balancer puppetlb mount the exported SSL directory from puppetca1, make it persistent in fstab and start Apache:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetlb~]$ mount puppetca1:/var/lib/puppet/ssl /var/lib/puppet/ssl | |
| [root@puppetlb~]$ /etc/init.d/apache2 start |
Repeat the above steps for puppetca2.example.com, ensuring that /var/lib/puppet/ssl is being synced from puppetca1 (using rsync or whatever other method you desire). Also it's not a bad idea to have a separate IP for the NFS export, that can float between puppetca1 and puppetca2 using heartbeat, or some other implementation of VRRP, so that the load balancer can re-mount the SSL directory in the event of the active CA server going offline. In fact I use NFS here just as a quick hack, in a production environment you might consider using iSCSI or a HA NAS. Also it's important to note that you don't really have to export the entire CA SSL dir from the CA to the LB. It's sufficient to just copy the CA cert, but not the private key, so the load balancer can verify the authenticity of connecting agents.
With SSL certificates, the server doesn't have a copy of the client's public key. So we need some other way to verify the client is who they say they are. This is where the third party comes into the picture. The third party (the CA) uses it's private key to digitally sign the public key of the client. This is a certificate. The CA's private key is not transferred to anyone, but the signed public key is transfered back.
When the client connects to a sever, it presents it's signed public key. The server uses the public key of the CA (NOT the private key) to verify the public key of the client is actually signed by the CA. At this point trust is established.
Configuring the Puppet Master nodes
Configuring the Puppet Master nodes is pretty much identical to how we configured the Puppet CA, with the exception that we don't have to deal with any certificates and exports, etc.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetmaster1~]$ echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list | |
| [root@puppetmaster1~]$ apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30 | |
| [root@puppetmaster1~]$ apt-get update | |
| [root@puppetmaster1~]$ apt-get install ruby libshadow-ruby1.8 | |
| [root@puppetmaster1~]$ apt-get install puppetmaster puppet facter | |
| [root@puppetmaster1~]$ /etc/init.d/puppetmaster stop | |
| [root@puppetmaster1~]$ update-rc.d -f puppetmaster remove | |
| [root@puppetmaster1~]$ vi /etc/puppet/puppet.conf | |
| [main] | |
| logdir=/var/log/puppet | |
| vardir=/var/lib/puppet | |
| ssldir=/var/lib/puppet/ssl | |
| rundir=/var/run/puppet | |
| factpath=$vardir/lib/facter | |
| templatedir=$confdir/templates | |
| [master] | |
| # These are needed when the puppetmaster is run by passenger | |
| # and can safely be removed if webrick is used. | |
| ssl_client_verify_header = HTTP_X_CLIENT_VERIFY | |
| ssl_client_header = HTTP_X_CLIENT_DN | |
| [root@puppetmaster1~]$ apt-get install apache2 libapache2-mod-passenger | |
| [root@puppetmaster1~]$ a2enmod passenger | |
| [root@puppetmaster1~]$ a2enmod proxy | |
| [root@puppetmaster1~]$ a2enmod proxy_balancer | |
| [root@puppetmaster1~]$ a2enmod proxy_http | |
| [root@puppetmaster1~]$ a2enmod headers | |
| [root@puppetmaster1~]$ vi /etc/apache2/mods-available/passenger.conf | |
| <IfModule mod_passenger.c> | |
| PassengerRoot /usr | |
| PassengerRuby /usr/bin/ruby | |
| # Recommended Passenger Configuration | |
| PassengerHighPerformance on | |
| PassengerUseGlobalQueue on | |
| # PassengerMaxPoolSize control number of application instances, | |
| # typically 1.5x the number of processor cores. | |
| PassengerMaxPoolSize 6 | |
| # Restart ruby process after handling specific number of request to resolve MRI memory leak. | |
| PassengerMaxRequests 4000 | |
| # Shutdown idle Passenger instances after 30 min. | |
| PassengerPoolIdleTime 1800 | |
| </IfModule> | |
| [root@puppetmaster1~]$ vi /etc/apache2/sites-available/puppetmaster.conf | |
| Listen 8140 | |
| <VirtualHost *:8140> | |
| SSLEngine off | |
| # Obtain Authentication Information from Client Request Headers | |
| SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1 | |
| SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1 | |
| RackAutoDetect On | |
| DocumentRoot /etc/puppet/rack/puppetmaster/public/ | |
| <Directory /etc/puppet/rack/puppetmaster/> | |
| Options None | |
| AllowOverride None | |
| Order allow,deny | |
| allow from all | |
| </Directory> | |
| </VirtualHost> | |
| [root@puppetmaster1~]$ cd /etc/apache2/sites-enabled | |
| [root@puppetmaster1~]$ a2ensite puppetmaster.conf | |
| [root@puppetmaster1~]$ mkdir -p /etc/puppet/rack/puppetmaster/{public,tmp} | |
| [root@puppetmaster1~]$ vi /etc/puppet/rack/puppetmaster/config.ru | |
| # a config.ru, for use with every rack-compatible webserver. | |
| $0 = "master" | |
| # if you want debugging: | |
| # ARGV << "--debug" | |
| ARGV << "--rack" | |
| require 'puppet/application/master' | |
| run Puppet::Application[:master].run | |
| # EOF /etc/puppet/rack/puppetmaster/config.ru | |
| [root@puppetmaster1~]$ chown puppet:puppet /etc/puppet/rack/puppetmaster/config.ru | |
| [root@puppetmaster1~]$ etc/init.d/apache2 restart |
Configuring the Puppet Agent nodes
Setting up the puppet agents requires that we install the puppet package.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetagent1~]$ echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list | |
| [root@puppetagent1~]$ apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30 | |
| [root@puppetagent1~]$ apt-get update | |
| [root@puppetagent1~]$ apt-get install puppet facter | |
| [root@puppetagent1~]$ vi /etc/puppet/puppet.conf | |
| [main] | |
| logdir=/var/log/puppet | |
| vardir=/var/lib/puppet | |
| ssldir=/var/lib/puppet/ssl | |
| rundir=/var/run/puppet | |
| factpath=$vardir/lib/facter | |
| templatedir=$confdir/templates | |
| [agent] | |
| server = puppetlb.example.com | |
| runinterval = 1800 | |
| splay = false | |
| configtimeout = 1200 | |
| noop = false | |
| autoflush = true | |
| report = false |
The server = puppetlb.example.com points the agent to the load balancer.
To request a CERT run:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetagent1~]$ puppetd -tv |
On the puppetca1 list and sign the cert:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ puppetca -l | |
| [root@puppetca1~]$ puppetca -s puppetagent1.example.com |
To verify that all is working, run tcpdump on all servers:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@puppetca1~]$ tcpdump -s 1024 -l -A port 8140 -i eth1 |
Then look for the validated headers:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ... | |
| Host: puppetlb.example.com:8140 | |
| Content-Type: application/x-www-form-urlencoded | |
| Accept: pson, yaml, b64_zlib_yaml, dot, raw | |
| X-SSL-Subject: /CN=puppetagent1.example.com | |
| X-Client-DN: /CN=puppetagent1.example.com | |
| X-Client-Verify: SUCCESS | |
| Via: 1.1 puppetlb.example.com:8140 | |
| X-Forwarded-For: 192.168.100.110 | |
| X-Forwarded-Host: puppetlb.example.com:8140 | |
| X-Forwarded-Server: puppetlb.example.com | |
| Connection: Keep-Alive | |
| Content-Length: 2815 | |
| ... |
