For many years it was my main work to reverse engineer software installation and configuration for hyper-scaled deployment automation and for OS provisioning for Windows.

Early on it was evident that for OS and software provisioning it was extremely important to be able to prove that you had documented the exact checklist of ordered steps to take a system from “pristine OS deployment” to “working configuration” as an input to solid automation code.

Over time, I developed proficiency in this type of reverse engineering, explored the hundreds of available free and commercial tools, blogged about it and eventually developed a business around multiple advanced training courses on reverse engineering Windows OS and software installation for the purposes of provisioning and deployment automation.

There is an entire eco system of commercial and free tools for system-wide diffing on Windows, so it was a bit shocking to recently rediscover that comprehensive system-wide configuration diffing tools for Linux are rare indeed.

Whether you are porting Windows reverse engineering skills to Linux, or are a Linux Engineer that has not experienced the productivity benefits of system-wide config diffing for reverse engineering - this post is for you ;)

Reversing a Working Raspian Configuration

For a robotics control project, I was recently setting up Raspian (the Raspberry Pi distro) to enable Bluetooth audio access via a running service. Bluetooth audio on Raspian needed a lot of work just to be functional. The requirement to simultaneously using the GPIO serial port to control an ardruino based robot created some knock-on complexity in figuring things out. Linux Bluetooth audio is also very per-user oriented (e.g. uses a user based systemd service and config files) because Bluetooth audio devices are frequently personal to users. However, it needed to work from the context of a systemd service with a special user logon so that the Bluetooth speaker could appear to give the robot commands.

The end result worked great - so I began the work to build “from scratch” instructions so others could do the same. However, the path to getting it working was extremely unclear due to the many changes (including unrelated ones) made along the way.

The last few steps to get Bluetooth audio working under a service remained unclear no matter how many things I tried or compared or repeated.

Devising a System-Wide Linux Config Diffing Solution

I mistakenly thought there would be a couple obvious solutions available to do a system-wide diff to quickly isolate the differences between the working and from scratch build.

I was unable to find any obvious options, but eventually found configsnap - created by the venerable Rackspace support team.

While it looked very promising, I hit additional snags - the help and documentatoin did not cover whether the concept of comparing two different machines was a valid use case and the repository docs did not show compare commands at all. Searching the Internet did not yield any how to documents or videos.

Whenever there is a gap this large (lack of linux system-wide diffing solutions in general and lack of how to information for the one I found) I can’t help but create a bit of Mission Impossible Code to bridge the gap.

The Mission Objectives and Parameters

 Mission Objectives and Parameters articulate the final objectives that emerged from both the preplanning and build process. Code Summary gives an out line of the code fragments. Code Call Outs highlights significant constraints, innovations and possible alternatives in the code.

1. Objective: Use tooling to find system-wide configuration differences between two seperate linux installations to quickly isolate the differences between a known good and non-working system.
2. Desirable Constraints In Meeting Objective:
   1. Origination Priorities:
      1. Source a ready-made solution, but if that fails…
      2. Assemble a solution from existing bits and pieces, but I don’t have the time to
      3. Build a solution from scratch.
   2. Leverage a “Zero Footprint” methodology where the use of the diffing tool does not create substantial configuration changes.
   3. Work on as many distros as possible (Configsnap is packaged for only some distros)

Code Summary

1. Using a Zero Footprint approach, use minimal code to bring down configsnap and run a before snapshot.

   1. If possible, enable “run from web”.

2. Perform a “known good” snapshot on the working reference system.

   1. Use a custom snapshot name that self-identifies its purpose (“crossmachinecompare”)
   2. Use a custom stage name that self-identifies its purpose (“knowngoodconfig”)
3. Provide sample commands for direct compare on the “compare-to” system.

Code Call Outs

Running Directly from Web

• The provided commands works around the fact that some systems block a directly downloaded script from being piped directly into bash - while a bit longer, this command works on a broader array of linux machines.

Zero Footprint

• Zero Footprint is a constraint that avoids changing any system-wide configuration to use code on a target system. Generally everything runs out of a directory and package managers are not used. This is more important for diffing utilities since they should not place resources that end up in the diff itself - especially if they are assessing production systems. In the past, I have had to create a zero footprint install of the CIS tool used to assess CIS Benchmark Hardening for both Windows and Linux.

Provided System-Wide ‘additional.conf’ Example

• By default the tool does not compare all of /etc/ nor any user configuration files. The example shows the most basic level of including these important configuration areas
• The example configuration provides an easy to extend example for refining the scope of comparison.

Complete Single Script Solution

• By creating it’s own configuration file and emitting the commands to use on “compare-to” target systems, the single script is fully self-contained and self-documenting.
• Single script solutions are frequently easier to automate since many management systems allow embedded transport of scripts (but not of packages, support binaries, etc).

The Code Itself

#Run this directly from this location with: curl https://gitlab.com/missionimpossiblecode/MissionImpossibleCode/-/raw/master/ConfigsnapCreateKnownGood.sh -O /tmp/ConfigsnapCreateKnownGood.sh ; sudo bash /tmp/ConfigsnapCreateKnownGood.sh

#Zerofootprint for both known good and compare-to systems - just delete /tmp/configsnap

if [[ -z "$(command -v python)" ]]; then 
  echo "Python must be installed and working, exiting..."
  echo "If you cannot install python on this or the compare-to system, read here about building it in an isolated directory: https://stackoverflow.com/a/42903156"
  exit 5
fi
mkdir -p /tmp/configsnap
curl https://raw.githubusercontent.com/rackerlabs/configsnap/master/configsnap -o /tmp/configsnap/configsnap
chmod +x /tmp/configsnap/configsnap

cat > /tmp/configsnap/additional.conf <<'EOF_CONFIG'
[allmachineconfig]
Type: directory
Directory: /etc/

[userconfigs]
Type: directory
Directory: /home/
File_Pattern: \..*
EOF_CONFIG

sudo ./configsnap --basedir=/tmp/configsnap/snaps --verbose --tag=crossmachinecompare --phase=knowngoodconfig

cat <<- EndOfMessage

Next Steps:

1. Sample scp command to pull this on a system to compare to: 
   scp -r user_on_this_system@thissystemdnsorip:/tmp/configsnap /tmp/configsnap
2. Sample auto-compare command on compare-to system:
   sudo /tmp/configsnap/configsnap --basedir=/tmp/configsnap/snaps --verbose --tag=crossmachinecompare --pre=knowngoodconfig --phase=post

To use as a known good snapshot managed in a centralized location, copy "/tmp/configsnap" to a shared location (or use git to commit to a repository) where you can pull it onto any system you wish to test for drift or changes.

To clean the zero footprint install from any systems, run "sudo rm -rf /tmp/configsnap"

EndOfMessage

Source Code for This Post

The code for this post is kept up to date and can be invoked directly from the web in this repository location: ConfigsnapCreateKnownGood.sh

Mission Impossible Code Series Inclusion

• The solution sticks to the Boring Technology selection criteria.
• The solution is implemented in a single script.
• The solution is Zero Footprint.
• The solution is portable between linux distros and comparison systems.

Solution Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

 The following content is a deep dive below the waterline into the nitty gritty details of how to take a similar approach to building solutions.

NOTE: You do not need this information to successfully leverage this solution.

What Does “<==>” Mean?

The notation “<==>”, which may contain logic like “<= AND =>” is an attempt to visually reflect the trade-offs inherent in using heuristics to commit to seleting a position on a spectrum of possibilities. By documenting these trade-offs below - the construction and serendipities of the final tuning are revealed. This seems to do at least three things for the consumer of this information:

1. You get to see the iceberg below the waterline of something I have built that I hope is “As simple as possible, but not simpler.” So you get to see why I claim that “The Creation of Simplicity is Necessarily a Complex Undertaking.”
2. You can more easily customize key parts of the solution to your liking, and not suffer from unintended consequences of those changes.
3. You can more easily apply this pattern to new problems that may be similar, but not identical.

Solution Architecture Heuristics

The overall solution is solving for “Use tooling to find system-wide configuration differences between two seperate linux installations to quickly isolate the differences between a known good and non-working system.”

Requirement: (Satisfied) Be Self Contained (Including Instructions)

• Mission Impossible Heuristic: Bring Everything You Depend On <= AND => Pack Light (Reference System)
• Reason: The more local dependencies a solution requires, the less portable it is and the more challenging it is to reuse across varying configurations. The gold standard is if the script can be used without instructions via the code containing embedded configuration and embedded instructions as needed.
• Coding Decisions:
  • Bring Everything: Use curl to download a raw copy of the python code from the repository - thereby avoiding dependencies on package managers (only a RHEL package exists) and Git (in the case of cloning the entire repo).
  • Bring Everything: Use configuration as code via a heredoc that creates the configuration file. This enables the entire solution to be in a single, run-from-web script. Configuration as code is also self-documenting by nature which avoids the need for external instructions.
  • Bring Everything: The Reference System code emits the instructions to be used on the Target Systems - providing User Instructions as Code also enables the self-contained, self-documenting nature of the solution.
  • Pack Light: While some few linux images (esp containers) may not include python, if it cannot be found, ask the user to resolve the dependency - with a hint on how to build python from source and run it without updating the entire system with Python.

Desirement: (Satisfied) Zero Footprint Approach

• Mission Impossible Heuristic: Leave No Trace Behind <= AND => Make Fingerprint Wipe-Down Easy

• Reasons:

  • Dependencies and system level configuration require permissions and soil the system. Drift detection on hardened systems (e.g. CIS Benchmark) or change management regulated systems (e.g. FDA Regulated) generally should not have their system level configuration or files changed .
  • The diffing tool itself must ensure that it’s own code, config and data do not become part of the comparison it is performing.

• Coding Decisions:

  • Use directory tree in /tmp.
  • Store Code, Config (additional.conf) and Snapshot Data in the same directory tree.
  • Cleanup is as easy as removing the root of the directory tree containing the code, config and data.

Serendipity: (Discovered) : Bring Everything You Depend On <= AND => Pack Light (Target System(s))

• Prior Requirement / Desirement: Be Self Contained, Zero Footprint Approach
• Prior Coding Decisions Result:
  • By storing Code, Config and Snapshot Data in the same directory heirarchy the resultant directory can be copied to a central location or repository or directly to a Target System and it is immediately runable with only two commands (even less prep than the Reference System)

Serendipity: (Discovered) Least Privilege Approach

• Prior Requirement / Desirement: Zero Footprint Approach
• Prior Coding Decisions Result:
  • By using /tmp and storing Code, Config and Snapshot Data, special permissions are only needed to run ‘Configsnap’ itself and then the operations of Configsnap only change a temporary, non-tracked area of the system.

Serendipity: (Discovered) Identical Configsnap Version, Configuration and Baseline Snapshot

• Diffing Heuristic: Two comparison targets should not have captured differences introduced by the comparison process.

• Reason: When diffing across two systems, the version of the diffing utility (Configsnap) and it’s configuration must be identical for results to be valid. This includes using older versions of the diffing utility even if they are no longer available from the original source.

• Prior Coding Decisions Result:

  • By storing Code, Config and Snapshot Data in one directory tree the version of all these components is frozen at the time point when the “Reference Snapshot” was taken.

Requirement: (Satisfied) Maximize Applicable Linux Systems This Can Be Used With

• Mission Impossible Heuristic: Optimize Your Choices <= AND => DeOptimize To Match The Breadth Of Required Scope
• Reasons: The usefulness of system-wide snapshots is applicable to all linux distros and architectures, ensure the proposed solutions reaches for this same scope.
• Coding Decisions:
  • Use curl to download a raw copy of the python code from the repository - thereby avoiding dependencies on package managers. Package managers complicate things because
    • The utility must have already been packaged for that package manager platform.
    • The utility must have a package per OS architecture (e.g. x86_64 and arm)
    • The package preparation must occur frequently enough to have the latest version of the software.
    • There are many different script commands to accomodate all possible package managers.