Infrastructure as Code, Part Three: Configuration With Salt

In part one of this miniseries, I introduced you to the concept of Infrastructure as Code (IaC) and explained that there are two broad categories of IaC tooling:

1. Orchestration tools that provision, organize, and manage infrastructure components.
2. Configuration management tools that install, update, and manage the software running on the infrastructure components.

In part two of this miniseries, I took a closer look at the orchestration tool we use at Crate.io: Terraform.

In this post, I round out the miniseries by taking a closer look at the configuration management tool that we use at Crate.io: Salt.

Salt is a tool that lets you automate the management and configuration of your infrastructure and the software installed on it. It is written in Python and maintained by SaltStack. Salt has two editions: the open-source edition and enterprise edition.

You can run Salt on many different operating systems, including various distributions of GNU/Linux, Microsoft Windows, and macOS.

Architecture

A Salt system is composed of two primary types of daemon:

1. Salt minions run on every server you want to manage with Salt.Minions are responsible for managing the configuration of software running on the same machine that they run on.

   Every machine you want to configure must have a minion running on it. If you have multiple minions running, Salt coordinates them, forming a distributed system.

2. Salt masters are responsible for coordinating the minions.Most Salt configurations use a single master. However, Salt supports running masterless minions as well as multi-master setups.

Here's an example setup:

(There is a third type of daemon, called a salt proxy minion. But that is out-of-scope for this introductory post.)

Salt minions are responsible for connecting to the Salt master. Minions receive jobs (i.e., instructions) via the open connection to port 4505, and send results (i.e., status updates) via the open connection to master 4506.

Salt follows the publish-subscribe pattern (pub-sub) for distributing jobs. That is, Salt masters publish jobs to all connected minions, and it is the responsibility of each minion itself to determine whether or not it is an intended target (i.e., whether or not to run the job).

Communication between master and minions is done using ZeroMQ, an asynchronous messaging library.

Core Concepts

Event-Driven Infrastructure

Salt enables you to develop an event-driven infrastructure.

What does that mean?

Well, you can configure Salt so that your infrastructure responds to changes (i.e., events) without your manual intervention.

For example, you could use a beacon to monitor a batch file and kick off a batch script to process the file any time a change is detected.

Or, maybe you want to monitor a service? You can configure Salt to watch the state of a running service and if the state changes (e.g., the service stops responding) you can automatically take some action (e.g., restart the service).

Remote Execution

Salt allows you to execute commands on one remote system or multiple remote systems at the same time.

Remote execution is done using execution modules, which are effectively just Python modules dedicated to this task.

Salt provides a decent built-in library of modules. You can also find a broad array of community maintained modules. And if none of these do what you want, you can write your own!

Configuration Management

In addition to remote execution, Salt allows you to manage the configuration state of your systems.

Configuration state can be managed using state modules, which are again effectively just Python modules dedicated to this task.

Each state module provides one or more functions that can be used to describe the desired state of a system. And these functions can be assembled into state files.

Similar to execution modules, you can write and incorporate your own. However, that will probably be unnecessary, since Salt already has a comprehensive selection of modules available for you to use.

State files are usually defined with YAML and have the .sls extension. However, state files are pre-processed by Jinja.

Jinja is a templating language that allows you to do variable interpolation and use simple logical constructs such as conditionals and loops. Using Jinja, you can gather and operate on data from the host system or from Salt pillars (see Pillars below) and dynamically generate the final state definition.

State files are, essentially, setup instructions, and they can be used to set up practically anything you can imagine.

Pre-written state files are called formulas. You can find lots of official and community maintained formulas on GitHub.

Grains and Pillars and Mines, Oh My!

There are three more concepts we should look at before continuing: Salt grains, Salt pillars, and the Salt mine.

These are Salt specific terms that are themed around salt the mineral. As in, the stuff you maybe sprinkle on your food.

Grains

A grain of salt is a small individual piece of mineral salt. By analogy, a Salt grain is a piece of information gathered by a Salt minion. This information typically relates to the underlying host system.

Grains are typically used to pass information from the minions to the master.

Grains are collected by minions when requested by the master. And grains are then cached by the master.

Pillars

A pillar of salt is a more complex structure made of mineral salt. Similarly, Salt pillars are collections of information, typically used to pass configuration values from the master to the minions.

The data in a pillar can then be accessed and used by your state files (see Configuration Management above).

The key distinction here is that your state files describe a generic strategy for setting up and configuring a particular aspect of your infrastructure. And then that strategy is further customized on a machine-by-machine basis by your Salt pillars.

Pillar data can be anything you want, but typical uses are for usernames, passwords, access keys, and so on. (It's worth noting here that pillars can be restricted so that only certain minions can access the data they contain.)

You don't need to use pillars. You could just use state files. But the modularization this approach enables can be really useful!

Like state files, Salt pillars are usually defined with YAML and also have the .sls file extension.

Pillars can be safely used for storing sensitive data (e.g., access keys) because pillar data assigned to a particular minion can only be accessed by that minion.

The Mine

A salt mine is a place from which mineral salt is extracted. By analogy, the Salt mine is a repository of arbitrary information returned by minions when executing commands as requested.

All minions can freely query data in the mine. However, only the most recent return values can are stored so this cannot be used for historical querying.

On Composability

Because Salt is configured using text files, it is common to keep those files in a version control repository such as a Git.

From a file system perspective, a basic Salt configuration typically has two primary top-level directories:

1. A salt directory where your state files (i.e., setup instructions) are defined.
2. A pillar directory where your Salt pillars (i.e., configuration values accessed by your state files) are defined.

At the top of each directory, there is a file, conveniently named top.sls.

The top file has three nested components:

1. An enumeration of environment names.Salt supports multiple environments. However, for the sake of simplicity, we will only consider Salt configurations with one environment: the base environment.
2. An enumeration of minion targets for that environment. These take the form of query strings.
3. An enumeration of files to apply to any minion in that environment matching the query string. Files are listed without their extension. So if the file is named common.sls, you would list it in the top file as common.

Here's an example top file:

base:
 '*':
   - common
'minion1':
   - users

This says:

• We have one environment: base.
• The common.sls file applies to any minion with an ID matching * (which is a glob expression matching all minions).
• The users.sls file applies to any minion with an ID matching minion1 (as this doesn't use any special characters, one and only one minion can ever match this pattern).

Recall that state files and pillars are both targeted in this way. This means we can apply any combination of setup instructions and configuration values we wish—to any machine that we want!

This is where the magic is, for me.

Using Salt, you can decompose your infrastructure setup and configuration into reusable components. These components can then be recomposed any way you want.

It can be a little tricky to get this right the first time. But once you've figured out the best approach for your situation, this modularization enables you to build a DRY infrastructure. And DRY code hopefully means faster development times and fewer issues.

Show and Tell

To anchor things a little, let's take a look at how to approach a hypothetical scenario.

Imagine you have three blank-slate machines: one master and two minions. All three machines are running Ubuntu. You're going to be deploying a Python application, which means installing Python dependencies with pip.

So the first problem you want to solve is how to install the same version of pip on every single machine.

How could you do this with Salt?

Recall that Salt pillars allow us to separate configuration values out from the instructions you provide to set up your infrastructure. You can use this to your advantage by configuring the version number of pip in its own file, pillar/versions.sls:

version:
  pip: 9.0.1-2.3~ubuntu1

To hook this pillar into your Salt configuration, you also need a top file, pillar/top.sls:

base:
  '*':
    - versions

Here, you're assigning the versions pillar to all minions using the glob syntax.

Now you have your configuration values specified, you can specify what Ubuntu packages to install with a state file, salt/common/packages.sls:

common_packages:
  pkg.installed:
    - pkgs:
      - git
      - vim
      - python-pip: 

Here, you're using the pkg state module to specify that you want the git, vim, and python-pip Ubuntu packages to be installed.

In this example, you are also passing the optional version number argument to the python-pip package. The double curly-braces {{...}} signify an expression in Jinja. So, in this instance, you are using Salt templating to substitute in the previously defined version number from your versions pillar.

Why use salt/common/packages.sls, though? Why not salt/common.sls, or salt/packages.sls? Well, it's likely you are going to want to have multiple state files, each one responsible for some component of your common setup. And with this directory structure, related state files can be easily modularized and kept together.

To make this work, we need an init file, salt/common/init.sls:

include:
  - common.packages

The init.sls file is sort of like a Python init file. It tells Salt that the common directory can be treated as though it was a single file, named salt/common.sls, and specifies which child files to include. (Read more about this in the docs.)

If you add a second state file to the common directory, you need to update the init.sls file accordingly.

Lastly, to hook the common directory into your Salt configuration, you will need a second top file, salt/top.sls:

base:
  '*':
    - common

Here, you're assigning the common directory to all minions using a glob expression.

Interacting With Salt

Okay. So far, we've looked at some of the basic Salt concepts, and we've looked at how to configure Salt.

The next step is interacting with Salt. And for that, you can use one of the command-line interface (CLI) tools that Salt provides. The primary tool is named salt.

Salt caches data. And so, a common reason to interact with Salt is to request an update.

For example, let's say you've updated the pip version number (see Show and Tell above). You can roll out this change by using the glob syntax to target all minions with the saltutil.refresh_pillar command, like so:

$ salt '*' saltutil.refresh_pillar

You can check to see what pillar data minions have access to, like so:

$ salt '*' pillar.items

Or, let's say you've updated your packages.sls file (see Show and Tell above). The latest state on the Salt master is known as the highstate. You can roll out this change by targeting all minions with the state.highstate command, like so:

$ salt '*' state.highstate

If you want to do a dry run before rolling out your changes, you can append a test=true argument. This will output the changes that would be made.

Caveats

If you have a large infrastructure setup (i.e., many minions) or a large team, it is possible for your Salt master (and in turn, the master branch of your configuration repository) to become a bottleneck.

This tends to be true of any system where multiple people are attempting to make concurrent changes to the same thing.

Bottleneck problems can be further exacerbated by lax rollout procedures that may result in different minions reflecting different versions of your master branch.

One approach is to monitor the state of minions and set up alerts for when they fall out of sync with the master branch for too long.

For example, at Crate.io, we use Prometheus for monitoring and altering. And we use saltstack_exporter for gathering metrics from Salt.

In addition to this precaution, you can mitigate problems by scheduling periodic highstates on all minions. Alternatively, you can integrate Salt into your Continuous Integration (CI) setup, so that changes made to the master branch are applied automatically.

Wrap Up

In this post, I introduced you to Salt, a configuration management tool that is flexible, scalable, and straightforward to use.

Salt is one of a family of tools that allows you to manage and develop your infrastructure using the same tools you use to develop code. This comes with the tremendous benefit of allowing you to decompose and recompose the components of your infrastructure configuration.

As with Terraform, Salt is a foundational IaC tool for us at Crate.io, and we are using it to develop our hosted CrateDB offering.

The official Salt documentation includes a beginner tutorial you can follow if you want to take things further. The beginner tutorial provides working code and walks you through the process of bringing up a Salt master with two minions.