Fully Automated, On Demand EBS Initialization in both Bash (for Linux) and PowerShell (for Windows)

Fully Automated, On Demand EBS Initialization in both Bash (for Linux) and PowerShell (for Windows)

There has been a nasty rumor going around that EBS volumes no longer need initialization (formerly pre-warming). The Amazon page that talks about this mentions that it is no longer needed, but that it IS needed for EBS volumes created from snapshots.

Although custom and Amazon AMIs are stored as snapshots, many people I talk to have come to believe that EBS volumes simply don’t need initialization, no matter what. I sought clarification from AWS support and learned that the boot volumes of our custom AMIs definitely need initialization as do AWS AMIs. Basically initialization is not needed if you just created a fresh EBS volume (console or Cloud Formation) that has never been snapshotted as part of an AMI.

The newest utility advised by Amazon is the File IO testing utility known as FIO. I decided to write a provisioning automation utility script that would automatically download this and run it or schedule it. Since I have been looking for excuses to improve my Bash coding skills - I decided to write the provisioning automation in both Bash and PowerShell.

Why Not PowerShell Core Darwin?

I am a big advocate of PowerShell Core 6. Over a year ago I wrote the code for the PluralSight course Securing AWS Using CIS Foundations Benchmark Security Standard so that it works in both Windows PowerShell and PowerShell Core for Linux. That code is here. I also wrote the current universal bash installer install-powershell.sh that is part of the PowerShell 6 repository to encourage experimentation with both the language and the development environment (vs code). Details here: http://pwsh.io/.

However, I decided to write the Linux version of this code in bash because:

  1. I enjoy, and frequently encounter the need for, writing code that has zero dependencies - runs on what ships with the OS with no extras. This is also why I didn’t pick a Python or Ruby which might not be on a barebones install or may need updating to support specific commands I need to use. (e.g. Ruby is no longer pre-installed on Amazon Linux 2)
  2. Requiring PowerShell just to run a disk initialization script is something a lot of pure Linux shops would not bother with.
  3. In my day job I support DevOps automation and provisioning tooling for both platforms and I wanted to push my Bash skills - including building a best practice snippet library.
  4. When I make comparisons of PowerShell and Bash, I want to be speaking from intimate knowledge of both. While I’ve had over 15 years at the Bash console - there is nothing like doing the same code in two languages to get to the next level of understanding.

What Capabilities Does The Solution Support?

This script solution supports on the fly installation of both the script and FIO. While I love run on demand solutions, I also like to build my solutions so that they can run offline by prestaging the script (and in this case FIO as well). Offline behavior is automatic when you copy the script local and make sure FIO is available ahead of time - either by a package install or by placing a copy in the current folder or in a folder on the path. The Windows auto install does not use Chocolatey (remember zero dependencies) - but I also packaged it on Chocolatey.org if you want to preinstall it using Chocolatey.

While building this solution a number of common provisioning automation scenarios came to mind - namely:

  • the possible need to continue with other automation while disk initialization is running (parallel execution)
  • the possible need for disk initialization to be executed AFTER the main automation so as not to steal any cycles from it (sequential execution)
  • the possible need for disk initialization to be certain to complete the full initialization of all volumes, even if it takes a very, very long time
  • the need for disk initialization to be reboot resilient

It turns out all of these common needs could be suited by a scheduled task / cron job which executes on a regular interval (but only allows one concurrently running instance of FIO). The scheduled task / cron job self-deletes upon completion of initialization. It is important to know that FIO does not pick up where it left off when restarting after a reboot - so if you have a lot of restarts or huge volumes - FIO will need a continuous uptime window long enough to allows it to read your largest device (if using the schedule capability, this only needs to eventually happen - you don’t have to stress about engineering a long enough uptime window into your automation).

The solution takes advantage of FIOs capability to initialize multiple devices in parallel and also it’s built-in CPU throttling.

The solution also kicks out the FIO disk report for each device to allow auditing of exactly what FIO did.

The solution also supports a broad set of Linux distros and PowerShell versions on Windows.

It supports quite a few additional capabilities - but I’ll leave that to the detailed list below.

Automated Solution Deployment

  • oneliner to download and run from GitHub.
  • on the fly install of FIO if not present (supports PowerShell 4,5.1 & 6.0.1 CentOS, RedHat, Ubuntu, Amazon Linux (1 & 2) other Linux distros will likely work. If you pre-install the distro matched edition of FIO on the system path, the script will detect it and not attempt an installation. This can also serve as a work around for distros for which there is no package manager package for FIO.
  • requires root or sudo - auto-detects what to use - errors if neither is available (Linux Only)
  • automatic complete offline operation by staging the script and FIO on a template image
  • defaults to prefer using FIO from path or current directory versus installing from web
  • setup recurrent scheduled task / cron job for (self preempts so only a single instance ever kicks off FIO):
    • reboot resilience - scheduled task is recurrent each x minutes, creates a flag file when done and self deletes after successful completion
    • future run - up to 59 minutes away (e.g. allow other automation to complete before starting disk initialization)
    • parallel run - allow automation to continue without waiting for FIO (set -RepeatIntervalMinutes 1 or -r 1)
    • reschedule by running with new value for -RepeatIntervalMinutes or -r
    • switch -unschedule or -u removes scheduled task

Flexible Operation

  • initialize multiple devices in parallel (default)
  • CPU throttling (nice)
  • automatically skips non-existence devices
  • takes stipulated device list (use -DeviceIDsToInitialize or -d)
  • when device list is not specified, enumerates all local, writable, non-removable devices (override incorrect device detection by specifying device list)
  • emits bare version (can be used to update or warn when a local copy is older than the latest online version)

Completion and Cleanup (when FIO runs to completion)

  • saves FIO output report
  • creates initialization done flag file - which preempts further runs and scheduling until done file is removed
  • removes scheduled task / cron job and copy of script when scheduled execution completes

Tested On

  • PowerShell 4 (Server 2012 R2)
  • PowerShell 5.1
  • PowerShell Core 6.0.1 (On Windows)
  • CentOS 7
  • Ubuntu 16.04
  • SuSE 4.4 Leap 42
  • Amazon Linux
  • Amazon Linux 2

Code For This Article

https://github.com/DarwinJS/DevOpsAutomationCode