Eric Radman : a Journal

Scripted Configuration

Systems administration can make use of many of the same concepts that software engineers employ: make a change, run to verify, then commit to deploy. Just like software development, this workflow depends a tight feedback loop, where we make one change at a time while monitoring for success or failure.

Unfortunately we have been have been tricked into believing that systems orchestration is an advanced topic. This is for good reason: traditional configuration management systems are cumbersome to use in trivial cases, and complex tasks quickly become intractable. The following demonstrations show how rset(1) can be used to define and test systems configuration. Since it is a tool for staging and executing scripts, you are free to use your own programming skills to build an approach tailored to the target environment.

Demo: A New Configuration for a Home Network

webm | mp4

This is the sequence of events:

0:06 The routes file defines the directories containing configuration to stage and a list of pln(5) files to evaluate for this host
0:20 The contents of a pln file are labels followed by tab-indented content. The default interpreter is /bin/sh
0:30 Create configuration files
0:50 The dry-run flag (-n) will show which hosts and labels match
0:58 Execute on hosts matching an address. The default regular expression for labels is ^[0-9a-z]. The built-in utilities rinstall(1) and rsub(1) are created locally so that they appear on the remote host
1:09 Subsequent commands will require elevated privileges since the ssh user is not root
1:09 Specifying the label pattern to run, a subset of the scripts. Since rset uses an SSH control master, there is very little performance penalty for writing small script fragments
1:49 rinstall returns exit code 0 if a file is installed or updated; this is used to conditionally reconfigure networking
1:49 Use the native tools already provided by the target OS to configure services

Demo: Manage a PostgreSQL Installation

webm | mp4

This is a moderately complex example of building a data-driven orchestration framework:

0:07 Starting with two newly provisioned VMs. These are referenced by IP address, but hostnames would work as well
0:15 Most of the code in this repository is written in ruby and it's behavior is driven by a yaml file which defines the parameters for each database server
0:37 Configure the first database (master)
1:00 The schema for this database is located in _sources directory, which is useful for large files. rinstall(1) fetches this over the built-in HTTP server
1:16 Upgrading from PG 11 to 12 accomplished by changing the version and running the configuration again
1:45 Configure the second database (standby)
2:23 Most of these scripts import common functionality from hostmap.rb
2:28 Each label contains logic for testing and establishing a consistent state on the remote system
3:06 The final step isn't configuration, just a convenient way to check on the replication status