Eric Radman : a Journal

Repeatable State with Salt

Executable Documentation

If you have stood up services and then tried to document the steps that you will realize that communicating the changes to others or your future self is not easy. At least it's not easy if you care about the documentation being correct or relatively complete. Salt Stack is an automation framework that works for large scale deployments, but also gives you a nice way to capture the configuration state of small-scale deployments such as your home firewall or personal web server. Salt has such readable configuration that it's like documentation that you can execute.


There are four commands that you need to know about

Command Description
salt-key Manage minion connections
salt Run rules by using salt master to push changes to the minions
salt-call Pull rules from the salt master and apply locally
salt-ssh Upload configuration and apply state over SSH

Pushing changes with SSH

Salt is popular for it's scalable master-minion model, but it can be used with SSH as well. salt-ssh reads a configuration file called Saltfile

  config_dir: ./etc

In the file etc/master we can then specify where the repository is (. for the current directory). state_verbose: False produces a detailed report only for changes.

        - .
state_verbose: False

There several downsides to using SSH. This mode is slow, and errors from the remote end (incomplete sudo configuration, missing python) tend to be obscure. /etc/roster specifies a list of hosts

  user: eradman
  sudo: True

Basic Configuration

To do a dry-run, use test=True with state.highstate or state.sls. The highstate method from the state module, applies all rules.

$ salt-ssh home state.highstate test=True

The rules are first defined in top.sls, which is a YAML file that maps hosts to rules.

  'workstation or':
    - openbsd-common
    - workstation
    - openbsd-common
    - home

The rule defined as openbsd-common can either be a YAML file called openbsd-common.sls or a directory containing openbsd-common/init.sls. The directory structure provides a logical place to put other files that you will reference, but that's up to you. intend to.

The name of each rule can be arbitrary

    - name: www
    - group: www
    - fullname: HTTP Server
    - home: /home/www
    - shell: /sbin/nologin

Or the name paramter will default to the rule name

    - group: www
    - fullname: HTTP Server
    - home: /home/www
    - shell: /sbin/nologin

This is a shortcut, but another implication is that if we wanted to apply more than one rule to the same target we need to give the rule a unique name.

The source path can refer to any location in this repo, this example expects a folder called files.


Even on a small scale this mode really shines because of it's speed. The configuration for the master consists of the following changes to /etc/salt/master

state_verbose: False
    - /srv/salt
    - /home/eradman/hg/config

Now the trick is to write a top file that allows you to run against a specific environment.

      - defaults

All service files in Salt are processed as Jinja2 templates, so {{saltenv}} will be replaced with the value of env on the command line

$ sudo salt '*' state.highstate test=True saltenv=$USER

As you might guess, $USER can be devel or any other path you define in file_roots.

Special variables such as passwords are normally retrieved using a pillar module. One such module is cmd_yaml, which simply expects a command with formatted output. This is from `/etc/salt/master`

  - cmd_yaml: lom_credentials.yaml

Now the values defined in this file can be rendered in templates, or viewed using

$ sudo salt $HOSTNAME pillar.items


Every time a template is rendered it has access to range of variables that pertian to the host it will be installed on. To see a list this list of values, run the grains.items rule

$ sudo salt $HOSTNAME grains.items

Now you can reference any of these values in your template

<VirtualHost *:80>
   ServerName {{ grains['localhost'] }}
   DocumentRoot /var/www/html

Since service files are processed as templates, you can build various

shortcuts by using loops
{% for interface in ["athn0", "bridge0", "em0", "gif0", "vether0"] %}
    - source: salt://home/hostname.{{interface}}
    - user: root
    - group: wheel
    - mode: 640
{% endfor %}

There are a number of built-in filters that you can employ them to remap all sorts of data. Here I'm using replace to map a paths to the naming convetion in my local repository

{% for file in ["miniupnpd.conf", "dhcpd.conf", "mail/smtpd.conf"] %}
    - source: salt://home/{{file|replace('/', '_')}}

Precheckin Validation

Salt doesn't provide a check-syntax flag, but several state modules provide a mock flag that enables you to process rules without hitting the network

#!/bin/sh -e
sudo salt-call state.highstate \
  --file-root=$(dirname $0) \
  --local --retcode-passthrough \

Running Salt's Development Branch

If you want to run salt's development branch or test a modification it is easy enough to install it to a virtualenv

  virtualenv-2.7 ~/local/saltenv
  . ~/local/saltenv/bin/activate

  cd ~/git/salt
  pip install -r requirements/dev_python27.txt
  pip install -r requirements/zeromq.txt
  ./ install

Now test changes by running locally

$ salt-call state.highstate --file-root=$PWD --local -l debug

salt-call would normally try to connect to a master, but --local instructs it to use to read configuration locally.

Example Configuration: WordPress

The following is a completely automated install of WordPress on OpenBSD. Unlike Ansible, Salt allows you to factor out components as you go. In this case I created a single file called vm/wordpress.sls that I included at the end of vm/init.sls

  - vm.wordpress

First we install the prerequisites for WordPress itself, including PHP and the libraries for accessing MySQL

    - pkgs:
      - mariadb-server
      - mariadb-client
      - py-mysql
      - py-pip
      - php-mysqli
    - skip_suggestions: true
    - fromrepo: pillar['openbsd_pkg_mirror']

OpenBSD supports multiple versions of PHP, none of which are configured by default. Symlink the default configurations like so

    - target: /etc/php-5.6.sample/mysqli.ini

    - target: /etc/php-5.6.sample/opcache.ini

    - target: /usr/local/share/examples/php-5.6/php.ini-development

Next initialize and start up a MariaDB server

    - creates: /var/mysql

Now that we have a MariaDB server we will create a mysql user and database for WordPress to use. In order to manage the new database Salt needs a MySQL library for Python. Salt can install packages via PIP itself as long as we symlink the pip to the version of Python we're using

    - target: /usr/local/bin/pip2.7

    - name: mysql
    - name: wordpress
    - name: wordpress
    - password: XYZ987
    - database: wordpress.*
    - grant: ALL PRIVILEGES
    - user: wordpress
    - host: 'localhost'

Now we can fetch and install WordPress itself. By specifying a path that should exist using creates we allow this to run only once. If we wish to wipe and reinstall we would simply rename the wordpress directory.

    - owner: www
    - group: www

    - name: 'wget && tar xvzf latest.tar.gz'
    - cwd: /var/www
    - creates: /var/www/wordpress/index.php
    - runas: www

WordPress needs to know the database connection parameters, which is set in wp-config.php

    - source: salt://vm/wp-config.php
    - owner: www
    - group: www

To establish FCGI communication to the service running PHP we can use file.replace to change only one line in the php-fpm configuration. This is a very nice option when you do not want to keep the entire file in sync

    - pattern: "listen = /var/www/run/php-fpm.sock"
    - repl: "listen ="
    - append_if_not_found: True

Finally we'll install the Nginx configuration and set the service to reload whenever nginx.conf changes

    - source: salt://vm/nginx.conf
    - user: root
    - group: wheel
    - mode: 644

    - watch:
      - file: /etc/nginx/nginx.conf

The section from nginx.conf that points to our new WordPress installation

  server {
      root /var/www/wordpress;
      index index.php;

      location ~ \.php$ {
          try_files $uri $uri/ =404;
          include fastcgi_params;
          fastcgi_index index.php;
          fastcgi_intercept_errors on;
          fastcgi_param SCRIPT_FILENAME wordpress$fastcgi_script_name;

      access_log   /var/www/logs/pineconeperfections.log main;

After putting all of these steps in order it makes it possible to see how the service operates and how the components relate.

Last updated on November 26, 2016