Thoughts on SaltStack after using it in production for a year!

I have been using SaltStack in production for around a year now. Making use of many of its features to simplify running several hundred Linux servers in a very small team (3 engineers) enabling us to continue to spend around 90% of our time on project work rather than support and fighting fires.

As this is one of my first posts on this blog ill divulge in a little history here to talk about our solution as a whole.

Solution prior to Saltstack.

Previously we had a spacewalk based solution,  this was great  for managing files. However these were static and clumsy often requiring different sets  of files for different machines. So while this did simplify where we had to manage files we didn’t really address the underlying issue of the complexity in managing files.

A change was needed…

Introducing Saltstack!

SaltStack is a configuration management tool similar to Chef or Puppet in core functionality.  Although I had previous experience with Chef I did choose a SaltStack as my configuration engine of choice. The reasons as to why were mainly due to its expandability and beautifully simplistic architecture, which opened up the doors to do anything.


Just what is it?

So we understand what SaltStack does but what is it? Well its a python based configuration tool and due to this allows countless expandability options. However more importantly and why I fell in love with saltstack is that its simply…

A giant message bus.

It’s a giant message bus with your whole infrastructure listening it and able to place messages on the bus. This, in combination with SaltStack’s orchestration engine, allows you to react to anything happening in your infrastructure in any possible way. Oh, a VM disk running out of space? Attempt to clean up and then use the hypervisor API to extend it if  needed. Constantly getting harassed to build a new cluster for Tom from  the development team? Create a chat based self-service platform leveraging the hypervisor API to create it for him.

The possibilities are quite literally endless.

Our Implementation.

Our salt implementation has 2 main parts. First and foremost, the star of the show, the salt-master. Who is constantly reconfiguring servers and changing our infrastructure. But with this much power we needed to know what changes had been made, when and why. As our infrastructure was turning more into infrastructure as code it only made sense to adopt some more developer like practices and thus Gitlab was chosen as a source code repository.

With Salts gitfs backend  functionality we could expose our git repository  as if it was a file-system mount. While great in concept we quickly found the 1 minute delay between us pushing config before we could run it in our test environment was just to much to handle. Luckily this was easily rectified by leveraging the Salt API to write to its message bus and trigger a state which updated the local filesystem.


Thoughts on the SaltStack community.

The SaltStack community and documentation is some of the best I have ever seen.

With a active IRC channel and documentation second to none its an awesome project and community to get involved in. I would highly recommend looking at involving SaltStack in pretty much any project you are working on.

Author: The Hacky Penguin

A DevOps Engineer with 5 years experience pushing companies towards Agile deployments, Infrastructure as code and container based solutions.