Installation d’un serveur Puppet V4 open source complet

Posted on by

Voici comment installer un serveur Puppet V4 (open source) complet comprenant:

  • Puppetserver
  • Puppetdb + Postgres
  • Activemq + Mcollective + plugins Mcollective
  • Hiera
  • R10k
  • Puppetboard
  • Puppetexplorer
  • Puppetdbquery
  • Modules usuels
  • Outils usuels

C’est l’équivalent d’un serveur AIO (All In One) de Puppet Enterprise.

A noter que l’on peut aussi se servir de ce tutoriel pour installer l’équivalent d’un serveur Split de Puppet Enterprise en répartissant les différentes briques ci-dessus sur plusieurs serveurs. Dans ce cas la répartition habituelle sera de ce type:

  • Serveur MOM (Master Of Master)
    • Puppetserver
    • CA
    • Activemq + serveur Mcollective
    • Hiera
    • R10k
  • Serveur Puppetdb
    • Puppetdb
    • Postgres
    • Client Mcollective
  • Serveur Console
    • Puppetboard
    • Client Mcollective
  • Serveur Master compilation
    • Puppetserver
    • Client Mcollective
    • Hiera
    • R10k

On peut installer plusieurs serveurs de compilation selon le nombre d’agent Puppet à gérer (horizontal scaling), même chose concernant puppetdb, pour postgres il faut passer en mode cluster et réplication.

Attention de bien spécifier tous les serveurs de compilation dans la directive « dns_alt_names » du puppet.conf du serveur MOM dans ce cas.

Prérequis: Disposer d’un serveur Linux de type RHEL/CENTOS 6.x avec au minimum 2 Go de RAM

 

Partie 1 : Puppetserver + CA

yum -y install https://yum.puppetlabs.com/puppetlabs-release-pc1-el-6.noarch.rpm

yum -y install puppet-agent puppetserver

cat > /etc/puppetlabs/puppet/puppet.conf << EOF
[master]
vardir = /opt/puppetlabs/server/data/puppetserver
logdir = /var/log/puppetlabs/puppetserver
rundir = /var/run/puppetlabs/puppetserver
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
codedir = /etc/puppetlabs/code
dns_alt_names = $(hostname),$(hostname -f),puppet

[main]
certname = $(hostname -f)
server = $(hostname -f)
EOF

chkconfig puppet on
chkconfig puppetserver on

/opt/puppetlabs/bin/puppet master --no-daemonize —verbose

Crtl ^C

 

Partie 2 : Puppetdb + Postgres

yum -y install puppetdb puppetdb-termini

/opt/puppetlabs/bin/puppetdb ssl-setup

chkconfig puppetdb on

cat > /etc/puppetlabs/puppet/routes.yaml << EOF
---
master:
facts:
terminus: puppetdb
cache: yaml
EOF

cat > /etc/puppetlabs/puppet/puppetdb.conf << EOF
[main]
server_urls = https://$(hostname -f):8081
EOF

yum -y install https://yum.postgresql.org/9.5/redhat/rhel-6-x86_64/pgdg-centos95-9.5-3.noarch.rpm

yum -y install postgresql95-server postgresql95 postgresql95-libs postgresql95-contrib puppetdb-terminus 

chkconfig postgresql-9.5 on
service postgresql-9.5 initdb

cat > /var/lib/pgsql/9.5/data/pg_hba.conf << EOF
# TYPE  DATABASE        USER            ADDRESS                 METHOD
# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
EOF

service postgresql-9.5 start

su - postgres
createuser -DRSP puppetdb
createdb -O puppetdb puppetdb

psql
\c puppetdb
CREATE EXTENSION pg_trgm;

exit

cat > /etc/puppetlabs/puppetdb/conf.d/database.ini << EOF
[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //127.0.0.1:5432/puppetdb
username = puppetdb
password = puppetdb
log-slow-statements = 10
EOF

cat > /etc/puppetlabs/puppetdb/conf.d/jetty.ini << EOF
[jetty]
# IP address or hostname to listen for clear-text HTTP. To avoid resolution
# issues, IP addresses are recommended over hostnames.
# Default is localhost.
host = $(hostname -f)

# Port to listen on for clear-text HTTP.
port = 8080

# The following are SSL specific settings. They can be configured
# automatically with the tool puppetdb ssl-setup, which is normally
# ran during package installation.

# IP address to listen on for HTTPS connections. Hostnames can also be used
# but are not recommended to avoid DNS resolution issues. To listen on all
# interfaces, use 0.0.0.0.
ssl-host = 0.0.0.0

# The port to listen on for HTTPS connections
ssl-port = 8081

# Private key path
ssl-key = /etc/puppetlabs/puppetdb/ssl/private.pem

# Public certificate path
ssl-cert = /etc/puppetlabs/puppetdb/ssl/public.pem

# Certificate authority path
ssl-ca-cert = /etc/puppetlabs/puppetdb/ssl/ca.pem

# Access logging configuration path. To turn off access logging
# comment out the line with access-log-config=...
access-log-config = /etc/puppetlabs/puppetdb/request-logging.xml
EOF

cat > /etc/puppetlabs/puppetdb/conf.d/config.ini << EOF
# See README.md for more thorough explanations of each section and
# option.

[global]
# Store mq/db data in a custom directory
vardir = /opt/puppetlabs/server/data/puppetdb

# Use an external logback config file
logging-config = /etc/puppetlabs/puppetdb/logback.xml

[command-processing]
# How many command-processing threads to use, defaults to (CPUs / 2)
# threads = 4

# Maximum amount of disk space (in MB) to allow for ActiveMQ persistent message storage
# store-usage = 102400

# Maximum amount of disk space (in MB) to allow for ActiveMQ temporary message storage
# temp-usage = 51200
EOF

cat > /etc/puppetlabs/puppet/puppet.conf << EOF
[master]
vardir = /opt/puppetlabs/server/data/puppetserver
logdir = /var/log/puppetlabs/puppetserver
rundir = /var/run/puppetlabs/puppetserver
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
codedir = /etc/puppetlabs/code
dns_alt_names = $(hostname),$(hostname -f),puppet

[main]
certname = $(hostname -f)
server = $(hostname -f)
pluginsync = true
storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb
EOF

chown -R puppet:puppet /etc/puppetlabs
chown -R puppetdb:puppetdb /etc/puppetlabs/puppetdb

service puppetdb restart
service puppetserver restart

 

Partie 3 : Activemq + Mcollective

yum -y install http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

yum -y install activemq *mcollective*

cat /etc/activemq/activemq.xml << EOF
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
  http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="$(hostname -f)" useJmx="true">
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>


        <plugins>
          <statisticsBrokerPlugin/>
          <simpleAuthenticationPlugin>
            <users>
              <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
              <authenticationUser username="admin" password="admin" groups="mcollective,admins,everyone"/>
            </users>
          </simpleAuthenticationPlugin>
          <authorizationPlugin>
            <map>

      <authorizationMap>
                <authorizationEntries>
                  <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
                  <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
                  <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
                  <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
                  <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
                </authorizationEntries>
              </authorizationMap>
            </map>
          </authorizationPlugin>
        </plugins>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="20 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="100 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
            <transportConnector name="stomp+nio" uri="stomp://0.0.0.0:61613"/>
        </transportConnectors>
    </broker>
</beans>
EOF

ln -s /usr/share/activemq/data /usr/share/activemq/activemq-data

chkconfig activemq on

service activemq start

cat > /etc/puppetlabs/mcollective/server.cfg << EOF
main_collective = mcollective
collectives = mcollective
libdir = /opt/puppetlabs/puppet/lib/ruby/vendor_ruby:/opt/puppetlabs/mcollective/plugins:/usr/share/mcollective/plugins:/usr/libexec/mcollective
logfile = /var/log/puppetlabs/mcollective.log
loglevel = info
daemonize = 1
direct_addressing = 0

# Plugins
securityprovider = psk

plugin.psk = unset

connector = activemq

plugin.activemq.pool.size = 1

plugin.activemq.pool.1.host = $(hostname -f)
plugin.activemq.pool.1.port = 61613
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = marionette

plugin.package.provider = puppet

# Facts
factsource = yaml
plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml

# Registration

registerinterval = 300

registration = Agentlist
EOF

chkconfig mcollective on

chown -R puppet:puppet /etc/puppetlabs/mcollective

/opt/puppetlabs/bin/gem install net-ping
ln -s /usr/libexec/mcollective/mcollective /opt/puppetlabs/mcollective/plugins/

echo "0,15,30,45 * * * * /opt/puppetlabs/bin/facter -py > /etc/puppetlabs/mcollective/facts.yaml" >> /var/spool/cron/root
service crond reload

service mcollective start

cat > ~/.mcollective << EOF
main_collective = mcollective
collectives = mcollective
libdir = /opt/puppetlabs/puppet/lib/ruby/vendor_ruby:/opt/puppetlabs/mcollective/plugins
logfile = /dev/null
loglevel = info

direct_addressing = 0

# Plugins
securityprovider = psk

plugin.psk = unset

plugin.package.provider = puppet

connector = activemq

plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = $(hostname -f)
plugin.activemq.pool.1.port = 61613
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = marionette

plugin.activemq.pool.1.ssl = false

# Facts
factsource = yaml
plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml

default_discovery_method = mc
EOF

mco ping

 

Partie 4 : Hiera

ln -s /etc/puppetlabs/puppet/hiera.yaml /etc

cat > /etc/hiera.yaml << EOF
---
:backends:
  - yaml

:hierarchy:
  - "hieradata/%{hostname}/%{module_name}"
  - "hieradata/%{hostname}/common"
  - "environments%{environment}/modules/%{module_name}/defaults"

:yaml:
  :datadir: /etc/puppetlabs/code/
EOF

puppetserver gem install hiera

chown -R puppet:puppet /etc/puppetlabs/puppet

 

Partie 5 : Modules usuels + liens symboliques vers les outils Puppet

# Autre solution pour l'installation de ces modules utiliser r10k (voir partie 6 et 7)
puppet module install puppetlabs/stdlib
puppet module install puppetlabs/concat

ln -s /opt/puppetlabs/puppet/bin/gem /opt/puppetlabs/bin/gem
ln -s /opt/puppetlabs/puppet/bin/augparse /opt/puppetlabs/bin/augparse
ln -s /opt/puppetlabs/puppet/bin/augtool /opt/puppetlabs/bin/augtool

 

Partie 6 : serveur git local pour r10k et définition des controls repos

adduser git
yum -y install git

mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys

ssh-keygen
cat ~/.ssh/id_rsa.pub > ~git/.ssh/authorized_keys

su - git
for i in environments.git hieradata.git manifests.git
  do
  mkdir $i  
  cd $i
  git init --bare
done

exit

for i in environments hieradata manifests
  do
  mkdir -p git/$i
done

cd environments
git init
cat > Puppetfile << EOF
mod 'puppetlabs/stdlib',
  :latest
mod 'puppetlabs/concat',
  :latest
mod 'apache',
  :local => true
mod 'epel',
  :local => true
mod 'puppetexplorer',
  :local => true
mod 'puppetboard',
  :local => true
mod 'python',
  :local => true
mod 'vcsrepo',
  :local => true
EOF

cat > environment.conf << EOF
modulepath = /etc/puppetlabs/code/environments/production/modules:$basemodulepath
manifest = /etc/puppetlabs/code/manifests
EOF

git add Puppetfile environment.conf
git commit -m 'initial commit'
git branch -m master production
git remote add origin git@git:~/environments.git
git push origin production

cd ../hieradata
git init
mkdir $(hostname)
echo "---" > $(hostname)/common.yaml
cat > $(hostname)/puppetexplorer.yaml << EOF
puppetexplorer::vhost_options:
  rewrites:
    - rewrite_rule:
      - '^/api/metrics/v1/mbeans/puppetlabs.puppetdb.query.population:type=default,name=(.*)$  https://%%{}{HTTP_HOST}/api/metrics/v1/mbeans/puppetlabs.puppetdb.population:name=$1 [R=301,L]'
EOF
git add $(hostname)/common.yaml $(hostname)/puppetexplorer.yaml
git commit -m 'initial commit'
git branch -m master hieradata
git remote add origin git@git:~/hieradata.git
git push origin hieradata

cd ../manifests
git init

cat > init.pp << EOF
## site.pp ##

File { backup => false }
EOF

cat > $(hostname).pp << EOF
# Configure Apache
class { 'apache':
  purge_configs => false,
  mpm_module    => 'prefork',
  default_vhost => true,
  default_mods  => false,
}

class { 'apache::mod::wsgi': }
class { 'apache::mod::proxy': }
class { 'apache::mod::dir': }

# Configure Puppetboard
class { 'puppetboard':
  manage_git        => 'latest',
  manage_virtualenv => 'latest',
  enable_catalog    => true,
  experimental      => true, 
}

# Configure Puppetexplorer
include puppetexplorer
EOF

git add $(hostname).pp init.pp
git commit -m 'initial commit'
git branch -m master manifests
git remote add origin git@git:~/manifests.git
git push origin manifests

 

Partie 7 : r10k

gem install r10k 

ln -s /opt/puppetlabs/puppet/bin/r10k /opt/puppetlabs/bin/ 

mkdir /etc/puppetlabs/r10k 

cat > /etc/puppetlabs/r10k/r10k.yaml << EOF 
cachedir: '/var/cache/r10k'

sources:
  environments:
    remote: 'git@git:~/environments.git'
    basedir: '/etc/puppetlabs/code/environments'
  manifests:
    remote: 'git@git:~/manifests.git'
    basedir: '/etc/puppetlabs/code'
  hieradata:
    remote: 'git@git:~/hieradata.git'
    basedir: '/etc/puppetlabs/code'

git:
  provider: shellgit

postrun: ['/bin/chown', '-R', 'puppet:puppet', '/etc/puppetlabs/code'
EOF

chown -R puppet:puppet /etc/puppetlabs/r10k 
 
r10k deploy environment -p

 

Partie 8 : Puppetboard (la configuration du module est visible en partie 6)

puppet module install puppet-puppetboard
puppet module install puppetlabs-apache

puppet agent -t

Puppetboard url : http://@ip/puppetboard

 

Partie 9 : Puppetexplorer (la configuration du module est visible en partie 6)

puppet module install spotify/puppetexplorer

puppet agent -t

Puppetexplore url: https://@ip/

 

Partie 10 : Puppetdbquery

wget https://raw.githubusercontent.com/f951753/puppetdbquery/master/puppetdbquery > /usr/local/bin/puppetdbquery
wget https://raw.githubusercontent.com/f951753/puppetdbquery/master/addreadonly2database > /usr/local/bin/addreadonly2database

chmod 755 /usr/local/bin/puppetdbquery /usr/local/bin/addreadonly2database

su - postgres
addreadonly2database
exit

puppetdbquery

 

Partie 11 : Outils usuels

gem install puppet-lint
gem install metadata-json-lint
gem install puppet-check
gem install r10kdiff

 

Un fichier « ova » est disponible ICI pour ce serveur AIO Puppet v4 tel que décrit ci-dessus.

NB: Attention actuellement, la dernière version de la librairie python Werkzeug (v0.12.1) casse Puppetboard, il faut revenir à la version v0.11.15 pour voir Puppetboard fonctionner… https://github.com/voxpupuli/puppetboard/issues/370 pour plus d’info.