Eric Radman : a Journal

Repeatable State with Salt

If you have stood up services and then tried to document the steps that you will realize that communicating the changes to others or your future self is not easy. At least it's not easy if you care about the documentation being correct or relatively complete. SaltStack is an automation framework that works for large scale deployments, but also gives you a nice way to capture the configuration state of small-scale deployments such as your home firewall or personal web server. Salt has such readable configuration that it can serve as executable documentation.


There are four commands that you need to know about

Command Description
salt-key Manage minion connections
salt Run rules by using salt master to push changes to the minions
salt-call Pull rules from the salt master and apply locally
salt-ssh Upload configuration and apply state over SSH

Pushing changes with SSH

Salt is popular for it's salable master-minion model, but it can be used with SSH as well. salt-ssh reads a configuration file called Saltfile

  config_dir: ./etc

In the file etc/master we can then specify where the repository is ( . for the current directory). state_verbose: False produces a detailed report only for changes.

        - .
state_verbose: False

There several downsides to using SSH. This mode is slow, and errors from the remote end (incomplete doas configuration, missing python) tend to be obscure. /etc/roster specifies a list of hosts

  user: eradman
  sudo: True

Basic Configuration

To do a dry-run, use test=True with state.highstate or state.sls. The highstate method from the state module, applies all rules.

$ salt-ssh home state.highstate test=True

The rules are first defined in top.sls , which is a YAML file that maps hosts to rules.

  'workstation or':
    - openbsd-common
    - workstation
    - openbsd-common
    - home

The rule defined as openbsd-common can either be a YAML file called openbsd-common.sls or a directory containing openbsd-common/init.sls. The directory structure provides a logical place to put other files that you will reference, but that's up to you.


Even on a small scale this mode really shines because of it's speed. The configuration for the master consists of the following changes to /etc/salt/master

state_verbose: False
    - /srv/salt
    - /home/eradman/hg/config

Now the trick is to write a top file that allows you to run against a specific environment.

      - defaults

All service files in Salt are processed as Jinja2 templates, so {{saltenv}} will be replaced with the value of env on the command line

$ doas salt '*' state.highstate test=True saltenv=$USER

As you might guess, $USER can be devel or any other path you define in file_roots.

Special variables such as passwords are normally retrieved using a pillar module. One such module is cmd_yaml, which simply expects a command with formatted output. This is from /etc/salt/master

  - cmd_yaml: cat /opt/lom_credentials.yaml

Now the values defined in this file can be rendered in templates, or viewed using

$ doas salt $HOSTNAME pillar.items

Handling Multiple Salt Environments

Out of the box Salt merges and munges all of the environments defined in file_roots, which is a disaster if you are using separate environments for development. And what else would you use multiple environments for? Set the Salt master to use only the environment you specified thusly

state_top_saltenv: base
top_file_merging_strategy: same

But we're still not done because Salt will mix up the pillar_roots we have configured if we don't disable pillar merging as well

pillarenv_from_saltenv: true
pillar_source_merging_strategy: none

Sadly we're still not done. If we don't specify pillarenv for each salt command pillar data will still be merged! We solve this with the following minion configuration

pillarenv: base

Now to check to see what SLS rules match a given minion by running

$ doas salt-call state.show_top

Similarly you hopefully have a consistent view of pillar data from each minion

$ doas salt-call pillar.items


Every time a template is rendered it has access to range of variables that pertain to the host it will be installed on. To see a list this list of values, run grains.items

$ doas salt $HOSTNAME grains.items

Now you can reference any of these values in your template

<VirtualHost *:80>
   ServerName {{ grains['localhost'] }}
   DocumentRoot /var/www/html

Since service files are processed as templates, you can build various shortcuts by using loops

{% for interface in ["athn0", "bridge0", "em0", "gif0", "vether0"] %}
    - source: salt://home/hostname.{{interface}}
    - user: root
    - group: wheel
    - mode: 640
{% endfor %}

There are a number built-in filters that you can employ them to remap all sorts of data. Here I'm using replace() to map a paths to the naming convention in my local repository

{% for file in ["miniupnpd.conf", "dhcpd.conf", "mail/smtpd.conf"] %}
    - source: salt://home/{{file|replace('/', '_')}}

Precheckin Validation

Salt doesn't provide a check-syntax flag, but several state modules provide a mock flag that enables you to process rules without hitting the network

#!/bin/sh -e
doas salt-call state.highstate \
  --file-root=$(dirname $0) \
  --local --retcode-passthrough \

Running Salt's Development Branch

If you want to run salt's development branch or test a modification it is easy enough to install it to a virtualenv

  virtualenv-2.7 ~/local/saltenv
  . ~/local/saltenv/bin/activate

  cd ~/git/salt
  pip install -r requirements/dev_python27.txt
  pip install -r requirements/zeromq.txt
  ./ install

Now test changes by running locally

$ salt-call state.highstate --file-root=$PWD --local -l debug

salt-call would normally try to connect to a master, but --local instructs it to use to read configuration locally.

Example Configuration: WordPress

The following is a completely automated install of WordPress on OpenBSD. Unlike Ansible , Salt allows you to factor out components as you go. In this case I created a single file called vm/wordpress.sls that I included at the end of vm/init.sls

  - vm.wordpress

First we install the prerequisites for WordPress itself, including PHP and the libraries for accessing MySQL

    - pkgs:
      - mariadb-server
      - mariadb-client
      - py-mysql
      - py-pip
      - php-mysqli

OpenBSD supports multiple versions of PHP, none of which are configured by default. Symlink the default configurations like so

    - target: /etc/php-7.2.sample/mysqli.ini

    - target: /etc/php-7.2.sample/opcache.ini

    - target: /usr/local/share/examples/php-7.2/php.ini-development

Next initialize and start up a MariaDB server

    - creates: /var/mysql

Now that we have a MariaDB server we will create a mysql user and database for WordPress to use. In order to manage the new database Salt needs a MySQL library for Python. Salt can install packages via PIP itself as long as we symlink the pip to the version of Python we're using

    - target: /usr/local/bin/pip2.7

    - name: mysql
    - name: wordpress
    - name: wordpress
    - password: XYZ987
    - database: wordpress.*
    - grant: ALL PRIVILEGES
    - user: wordpress
    - host: 'localhost'

Now we can fetch and install WordPress itself. By specifying a path that should exist using creates we allow this to run only once. If we wish to wipe and reinstall we would simply rename the wordpress directory.

    - owner: www
    - group: www

    - name: 'wget && tar xvzf latest.tar.gz'
    - cwd: /var/www
    - creates: /var/www/wordpress/index.php
    - runas: www

WordPress needs to know the database connection parameters, which is set in wp-config.php

    - source: salt://vm/wp-config.php
    - owner: www
    - group: www

On OpenBSD php-fpm will To establish communication on a local socket by default. We just need to provide the path, which is relative to the chroot in

    - source: salt://vm/httpd.conf.conf
    - user: root
    - group: wheel

And restart whenever the config changes

    - watch:
      - file: /etc/httpd.conf

Where httpd.conf contains

server "" {
        listen on "*" port 80
        log style combined
        alias ""
        root "wordpress"
        directory index index.php
        location "*.php" {
                fastcgi socket "/run/php-fpm.sock"

Since php-fpm is running in a chroot the child process talking on the local socket will not be able to resolve hostnames unless we install resolv.conf. Notice that the source is not our salt repo, but another local file on disk

    - source: /etc/resolv.conf
    - user: www
    - group: www
    - makedirs: True

Last updated on November 29, 2019