In a previous life, I was on a team that reviewed the IAM policies specified by developers when they created new Cloud applications or required additional permissions for application updates that used new cloud features on AWS. The review was for both Infrastructure as Code automation for deployment as well as for Application runtime operations.

This task generally falls to the developer of the code because only they know the code operation well enough to ensure all code-paths are accommodated in least privilege discovery and testing during security engineering.

While reviewing the developer-authored permissions was very intense - the process developers used to determine the permissions was more intensive. A common methodology is to lock down the application or automation runtime permissions and go from failure to failure to discover the minimum permissions that are needed. This is incredibly labor intensive and on very large application or automation code bases it can add weeks to the release cycle. There is a much better way to do this that is much more productive and nearly eliminates deep rabbit hole explorations on permission dependencies in complex code.

A Well Kept Secret

AWS IAM Access Analyzer (Least Privilege) Policy Generator is already one of the best-kept secrets for AWS. The most tedious way to discover the least privileges for an application or infrastructure as code is to lock down a profile and work your way through all the failures one by one. There are multiple problems with the approach:

• The sheer number of hours.

• Many situations where deep dependencies between permissions make it difficult to understand exactly what permission is needed. This can lead to many failure iterations to isolate a single permission.

• This is tough enough on small stacks, but at the level of production scaling stacks I've worked with - it becomes impossible.

• This task is generally pushed to Application Developers in the case of applications - but it is not only out of many of their daily proficiencies - it is a very specialized area of cloud computing - permissions.

A Better Approach

If one could log EVERY permissions request, a much better approach is to give an application admin permissions, run it to exercise its capabilities and then aggregate the log information. There are many 3rd party scripts and utilities to do this for various permission systems. However, in AWS, there is a specific feature known as "Policy Generation" within the IAM Access Analyzer service. It analyzes CloudTrail logs (which have records for every required permission) to determine the least amount of permissions required for a given application or automation to run. The catch is the documentation and all blogs I've seen before always talk about using it against an IAM Role. This makes the setup more challenging because, unless you've done it before, running your application or automation using a role may require new skills.

An Even Better Kept Secret

However, IAM Access Analyzer has an even better-kept secret hidden within. Someone can correct me if I'm wrong, but I cannot find AWS documentation, blogs or videos on the following nuance of this feature.

When you are viewing an IAM User, clicking the "Generate policy" button in that user profile allows IAM Access Analyzer Policy Generator to use a USER ID to analyze the logs. This is very useful because using AWS Keys to exercise the application or automation is familiar to many more code developers - this means that one does not need to be as much of an AWS security specialist to leverage this automated security engineering feature. Using keys is also sufficient for security engineering since you can delete the keys or the entire user after doing your engineering.

Generate Policy button at the bottom of the default Permissions tab in an IAM user.

Practices to Consider

Additional good practices for using IAM Access Analyzer include:

• Making it Comprehensive

  • Since IAM Access Analyzer is reverse engineering from log data - the entire application or automation must be exercised in order to discover all permissions.

  • For automation it can be easy to forget that it may create alternative infrastructure based on input parameters.

  • Since access analyzer takes a time range, you can run the automation many times to surface all the required permissions.

  • It may be prudent to create a test plan to ensure nothing is missed.

  • For Infrastructure as Code, the Initial run must be captured because IaC reports "true" for things that are already configured as desired ("Desired State") and does not attempt to configure them - so there will be no CloudTrail log data for things that are skipped due to already being configured.

  • For Infrastructure as Code, the teardown must also be captured if clean removal of resources with the least privilege permissions is desired.

• Making it Faster:

  • Create a new role or user for the purpose of least privilege discovery so that there will be no other activity on the user id or role. This helps ensure that unneeded permissions are not added to the least privilege permissions since there simply isn't any unrelated activity to track.

  • Consider creating a new Cloud Trail to also limit the amount of data that will need to be analyzed. This also allows you to analyze account level data even if your organization has Cloud Trail implemented and collecting to a bucket the local account does not have access to. Don't forget to disable or delete it when done to prevent duplicate data collection.

  • When running the Access Analyzer you can also bound it by region that it examines - so if the entire application or automation runs can be in a single region, the data analysis will go even faster.

• Making it Cleaner:

  • If you created a dedicated Cloud Trail, when done, deconfigure it and delete the bucket that houses the data.

  • When done, delete either the KEYS or the entire User or Role that was used for least privilege discovery.

Variation for Working Examples and Templates

The tightest least privilege IAM permissions are coded to specific resources. However, if your automation needs to handle many regions or many naming schemes, then you may need to replace the generated policy's variables with '*', these include any references with curly braces such as ${region} and ${account}.

Documentation and References

The following do not cover the user id functionality as far as I can tell:

• IAM Access Analyzer (Least Privilege) Policy Generator

• How to use IAM Access Analyzer policy generation | Amazon Web Services

Photo by Christina Morillo

You know what I mean, it generally looks something like this when redacted:

Fortunately, there's a simple solution for hiding the AWS account information in screenshots and videos. Ad blockers that use filter lists like AdBlock Plus can be configured with a custom rule to automatically remove the account ID span. Here is the AdBlock formatted rule that will hide the information.

[Adblock Plus 2.0]
! Version:
! Title: CustomList
! Last modified:
! Expires:
! Homepage:
!
console.aws.amazon.com##span[data-testid="awsc-nav-account-menu-button"]

Note: Instead of trying to manually configure this file or a rule, see the section below "Direct Sourcing by URL"

So it will now display like this:

The custom ad blocker rule obscures AWS account IDs in screenshots, but still allows easy access to view them when necessary. The account information drop-down arrow remains visible and clicking it reveals the account number, your current IAM user and all the regular links that are normally on this menu.

AdBlock Extensions - Disable "Acceptable Ads"

If you are using either the AdBlock or AdBlock Plus extension, you will likely need to disable the built-in list "Acceptable Ads" as discussed in this AdBlocks Support Solution. When I did not disable this list, the AWS Account information would continue to be visible.

Built In Ad Blocker in Vivaldi

I generally use Vivaldi which supports AdBlocker rules natively. Since it does not depend on third-party browser extensions to remove the AWS Account data display, it does not open one up to additional security implications of third-party extensions that have page-level access to your AWS console. It also is not configured with the "Acceptable Ads" list.

Direct Sourcing By URL

Most ad blockers can be configured with an import URL. If you would like to import this rule and any improvements to it in the future, you can import from this URL: https://gitlab.com/missionimpossiblecode/adblock-for-tutorials/-/raw/main/adblockconfig.txt

Here are the instructions for configuring filter lists for AdBlock.

On the flip side, AWS CloudShell cannot include every possible utility you may need or want to use. AWS CloudShell is currently based on Amazon Linux 2 and so the relevant commands apply.

Know Before You Install

Here are some tips to make note of when considerating doing installations and configurations within AWS CloudShell:

• AWS CloudShell is based on Amazon Linux, which means you must use yum. When you are reading installation documentation you want to look at the yum commands. The Amazon Linux yum repos are on by default.

• Since it is Amazon Linux, favor installing from Amazon Linux Extras if the software you are looking for is available there. These extras instructions also tell how to add the RHEL extras repo to Amazone Linux.

• Check if what you want is already installed on this list of AWS CloudShell Pre-installed Software.

• AWS CloudShell allows passwordless sudo by default - so using sudo to install things is easy.

• Making installations idempotent makes sure they are quick and error free. This essentially means checking if something is already present and only installing or configuring it if it is missing.

• AWS CloudShell does not persist configuration changes after you log off of the AWS account. They are also remembered on a per-account, per-region basis - so an installation into CloudShell while viewing us-east-1 is not retained if you switch to eu-central-1. This really means that at any given time you use CloudShell it will not be likely to have the configuration you need.

• For data storage (not installs and configuration) the $HOME folder can store up to 1GB of data per-region and is deleted after 120 days from the end of the last session.

• The $HOME directory will give a disk error at 1GB. A workaround is to use sudo in the /home directory for commands that will download more than 1GB. This content will not be retained, but if it is just temporary dependencies (like with complex terraform stacks) then at least you can get the work done.

• With regard to Kubernetes administration, the automatic kubeconfig setup using AWS CLI works great in AWS CloudShell. The code below includes config-kubectl.sh which do this configuration for you. It will enumerate the EKS clusters and prompt for one if you do not provide one.

Run From Web To The Rescue

For some years now I've been using what I call "Run From Web" code in both Bash and PowerShell to enable anyone to quickly use scripted solutions directly from git without a lot of fuss. This makes them ideal for helping with the fact that AWS CloudShell forgets configuration changes on logoff and only remembers them on a per-account, per-region basis in the first place.

The specific scripts also check if something is installed and don't install it if already present, making them lightning fast when doing incremental configuration.

Code Review

Below is the code block for installing terraform, notice:

• It self-documents how to run it from the web. (including hard lessons learned about not piping downloaded scripts directly into bash.)

• The if statement ensures we only do work if we need to. Also if you run this terraform configuration and then later run the add-all.sh script - which also includes terraform - the installs layer up without duplicate work or errors (idempotency).

• The final echo command tells us the installed version. It functions as both a test of whether an install worked as well as a visual confirmation that something is already installed when no install was required.

# Run this directly from the web with:
# curl -sSL https://gitlab.com/guided-explorations/aws/aws-cloudshell-configs/-/raw/main/add-terraform.sh -o $HOME/add-terraform.sh; chmod +x $HOME/add-terraform.sh; bash $HOME/add-terraform.sh

echo "AWS Cloudshell Configuration"

if [[ -z $(command -v terraform) ]]; then
  echo "adding terraform"
  sudo yum install -y yum-utils 
  sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
  sudo yum -y install terraform
fi 
echo "TERRAFORM VERSION: $(terraform --version)"

Managing Terraform Templates and State

One of the main things I will be using AWS CloudShell for is running the Terraform templates of EKS Blueprints. These are likely to have more than 1GB of downloading dependencies, so operating in the $HOME directory of AWS CloudShell causes an out of disk space error. However, operating anywhere else causes the directory to be tossed as well as introduces some challenges with constantly use sudo to be able to change the disk.

The work around is to create a root directory and permission it for the cloud shell user and then user the -state switch of terraform to store the state in $HOME where it will persist for up to 120 days.

The below code repository includes prep-for-terraform.sh that will prepare the setup for you and shows commented terraform commands to go with a default setup.

Code For This Article

Here is the open source code: AWS CloudShell Run From Web Configuration Scripts

Mission Impossible Code Series Inclusion

• The solution is concise.

• The solution is idempotent (only takes steps necessary when things are missing).

• The solution runs directly from the web.

Checklist Authoring, Navigation and Completion without Dual-Pain Markdown Editors

“Dual-pain” is not misspelled here — while developers may consider a live rendering window an upgrade to coding in raw markdown, it is very cumbersome for those who do not have a developer role, but need to author Template Checklists or complete Tracking Checklists.

The interactivity of checklists on GitHub and GitLab should generally be used for Tracking Checklists as they facilitate collaboration through immediate, shared updates and do not require Git commits or merges during real-time operational events.

There may be times when visual, desktop-based utilities offer advantages over this model, such as:

• For easy authoring of checklists.
• For navigation of complex checklists In an outline mode while completing them.
• For completing visual markdown structures that are not interactive on GitHub and GitLab (such as fill-in fields or tables).
• For implementation of more elaborate “completed marking” (for example, adding date-time stamps and/or colors).

Typora + CopyQ

Combining Typora and CopyQ expands checklist execution and authoring options. CopyQ allows the insertion of standardized markdown snippets using hotkeys and Typora can directly receive markdown and immediately renders it. This allows adding the following additional completion marking capabilities:

• Apply Markdown formatting completion mark (such as bold).
• Insert the current date and time as part of “Completion marking.”
• Prompt for information and include it as part of the insertion text. In the provided CopyQ code there are examples of prompting for both Template Checklist authoring and Tracking Checklist execution.
• Apply HTML or CSS formatting — for instance, colored text background. HTML and CSS formatting render nicely in Typora, but not in GitLab and GitHub.

Cumbersome Markdown Editing Eliminated

In the past I have had my favorite code editor (VS Code) loaded up with many markdown extensions — each one completed different parts of the puzzle and did it in different ways. Sometimes they conflicted or even prevented editing the markdown by no longer recognizing specific keystrokes. I have removed those plugins and instead run a plugin that simply opens a file type in my operating system’s default application for markdown files and configure Typora to handle markdown extensions in the operating system.

Recording Information in Checklists and Cognitive Loading

The elimination of cognitive load during a stressful activity, such as managing IT changes, is also a huge benefit. No matter how sharp you are, humans have limits as to how many things they can juggle in any situation — but the sheer risk of production changes saps even some of that capability away because we are being extra vigilant to ensure everything is done just as planned. Under these conditions, getting some markdown code wrong or having to keep recalling the need to properly format can be very distracting.

A primary way around this dilemma is to prepare a copy of a template checklist in advance of the change event to the degree that it is possible. Filling in versions, titles, change dates can usually be done a few days ahead of time. Typora can be used for this by simply copying the body portion of an Issue or Merge/Pull Request in, and when done, pasting it over the body content when done editing.

If your checklist execution procedures require the insertion of screen captures, log data or inserting names and dates and versions — then editing the markdown will be required and this is where Typora will shine by making it nearly as easy as using Google Docs or your favorite word processor. Using Typora for execution does not allow easy real-time collaboration — so there is a potential trade-off if the checklist is used collaboratively.

Multiplatform Developer Workstations for DevOps

Across my professional and personal projects, I use all of Windows 10, Mac OS (Catalina) and LinuxMint for editing code and markdown. I use Visual Studio Code on all of them as my primary text editor. These operating systems are also common across DevOps teams — many times the same team will have at least two of these.

Typora and CopyQ both run on all three OSes as well and provide a consistent user experience on all platforms.

Cost of This Solution

Paid for checklist SaaS solutions can represent a lot of adoption friction for team tooling — not just because of cost, but due to:

• Establishing new system users in a third-party system.
• Having important audit records in the care of yet another third-party.
• Can not integrate directly with the built-in checklist interactivity of GitLab and GitHub.

Typora is $15 lifetime for 3 devices and CopyQ are free and there are no other elements to the solution that carries a cost. The package manager for your OS may have the last free version of Typora (0.11.18) available - but I have to say that at $15 Typora pays for itself in many times over. For instance, without Typora I don't bother doing markdown tables at all and with it, they are a breeze.

CopyQ for Standard Markdown Checklist Snippets

If you have team members or teams who do not like Typora, CopyQ can provide value by itself in that a standardized set of Markdown insertions can be defined and shared via a CopyQ configuration text file in Git. This can allow the establishment of some conventions for authoring and executing checks regardless of the editor in use.

Per-Step Time-Date Stamps on “Done Marking” Required

Time-date stamps on “done marking” may be desirable or required by your checklist execution methodology. Some checklists may be multiday or long-running — even if completed at one time. Time-date stamps add value by creating a historical time record of completion time and elapsed time — which allows for predicting ops events duration and event correlation if there are problems with a change.

As mentioned earlier, GitLab’s markdown interactivity does provide these stamps in the discussion log — however, if you ever had to audit a checkbox for completion or timing — you must review the entire log for all toggles of the checkbox in question to get the information. In addition, if you have checkboxes that have the same text repeated, it will be difficult to sort out which ones are being manipulated in the audit log.

When to Use Typora Over Other Editors or Web Editors

• For authoring checklists — even when they are stored as issue or merge/pull request templates.
• When advanced formatting is desired or required — you will be more willing to author helpful constructs such as tables and inserting graphics. Inserting graphics using Typora is especially compatible with git by allowing you to seamlessly copy the inserted graphic to the same directory as the markdown where you are inserting it.
• For completing checklists that have a lot of data recording or where more sophisticated done marking is required. This inhibits real-time Collaboration on the document because you have to go into edit mode in the issue or merge/pull request.

Working Example of What Is Possible with Typora and CopyQ (with Code)

The premade stamps provided here all have CSS colors — but also use standard markdown bold so that they stand out reasonably well with or without CSS rendering.

Setting Up Typora and/or CopyQ

The following instructions have been tested on Windows 10, OSX and Linux (LinuxMint 19.x).

Installation on Windows (Chocolatey)

Installation on Mac (Brew)

Installation on Linux (Ubuntu)

Typora is only tested on Ubuntu.

Here are the Linux installation instructions: http://support.typora.io/Typora-on-Linux/

CopyQ in public package repos is published as “CopyQ”: https://copyq.readthedocs.io/en/latest/installation.html

Configure Typora

These settings are optional, but helpful for the concept of Markdown Checklists. Access these settings from File => Preference:

**Images Insert** => “Copy image to current folder (./)”
**Save & Recover** => “Auto Save”
**Privacy** => “Send Anonymous Usage Info”
Configure CopyQ with TODO, DONE, SKIPPED and RECORD HERE Stamps

The following snippets give you a minimum to start with and also serve as examples to customize for your own work. Using the technique of distributing your own command ini allows you to standardize across your team.

While we are pairing CopyQ with Typora, it works fine with any Markdown editor.

Import this CopyQ settings import file: https://gitlab.com/darwinjs-ideas/GitOpsforHumanProcessedChecklists/-/blob/master/copyq-command-import.ini

1. Start CopyQ from the menu and it becomes memory resident.
2. If the UI does not show, open CopyQ from your “tray” area by clicking the icon and selecting “Show/Hide.”
3. In the CopyQ UI, Open “File => Commands / Global Shortcuts.”
4. Use the “Load Commands” button to load the INI.

Here are the hotkeys. They render with background colors in Typora (renders CSS) and without the background color on websites like Github and Gitlab. The bold and square brackets make the stamps recognizable when colors are not rendered.

• ALT-SHIFT-S: **[TODO]**
• ALT-SHIFT-D: **[DONE 07/28 @ 10:42]**
• ALT-SHIFT-Z: **[SKIPPED 07/28 @ 10:42 ]**
• ALT-SHIFT-X: **[RECORD RESULT HERE]**
• ALT-SHIFT-R: _REPLACE_WITH_ <whatever was typed in response to the prompt>

Tested On

• Windows 10
• MacOS
• LinuxMint 19.1

Conclusion

This article series has set out to communicate some foundational ideas about managing DevOps procedures in a GitOps world.

First, we established that the “human code” we find in our DevOps solutions is just as critical as computer code and therefore belongs in git.

We went on to claim that “human code” can be recognized because it takes the form of checklists, which reassemble computer code in that they are exacting, ordered and sometimes conditional instructions for accomplishing the desired outcome.

We touched on the evidence-backed claims in the highly recommended book “The Checklist Manifesto” that the helpfulness and necessity of checklists positively affect all individuals and teams no matter how smart they might be.

The templating and interactive checkbox features of GitHub and GitLab were highlighted as critical enablers for integrating of checklists into GitOps software change-management workflows and serve as working example repositories on both were also provided.

Finally, we discussed some rich multiplatform desktop tools to help with ease of editing and standardization of Template Checklist creation and Tracking Checklist completion. Working code examples for configuring CopyQ were also provided.

It is my hope that this three-part deep dive will help you, your team and your organization to experience the massive quality improvements that the simple concept of collaborative checklists has to offer.

Code, Examples and Architecture for This Article Series

“GitOps for Human Processed Checklists (Part 1)”

“GitOps for Human-Processed Code (Part 2): Interactive Markdown Checklists on GitHub and GitLab”

CopyQ command import file for stamps discussed in this article.

Product Selection Architecture Map For This Solution

“Architecture Heuristics for This Solution: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives”

GitLab Sample Repository

GitHub Sample Repository

“The Checklist Manifesto”

• The checklists in question are for highly routinized procedures that everyone should implicitly know and execute automatically.
• The frequency of measured mistake avoidance by using checklists for these standardized procedures is very significant.
• The very professionals who are considered the sharpest in the room (lead surgeons, airplane pilot captains, etc.) were not immune to needing checklists as many had assumed they might be.
• Most complex activities are tackled by a team — many breakdowns in proper outcomes come from assumptions about what others have done or who has the right to point out problems during a procedure. Checklists resolve this because they ensure communication and give anyone the right to point out missed items.

The “Checklist Manifesto” demonstrates that checklists don’t just prevent faults due to known issues — they also shift the human relational and communication dynamics of the team so that the can consistently produce better outcomes. So, if anyone ever tells you they don’t need a checklist because what they are doing “ain’t brain surgery,” you can let them know real brain surgeons actually use checklists for the easy stuff that also has life-threatening consequences if mistakes are made.

Full Checklist Lifecycle

There is not a standard set of terminology for the checklist lifecycle, so we’ll apply a little of our own and tack it toward DevOps checklists with the terms “Template Checklist” and “Tracking Checklist.” As you’ll see, the important differences between these checklist types has implications for what tooling is used to create and update template checklists versus tooling for processing tracking checklists.

What we will call a “Template Checklist” is where we express the following:

• What needs to be done.
• Required order of step completion and/or parallel step execution allowances.
• How to complete procedures (or pointers to procedures if too long to be contained inline).
• Why a specific approach or order is important (to facilitate procedure adjustments if problems with the standard procedure are experienced).
• Fill-in fields that may need to be tailored before the checklist starts or field engineering tracking checklist completion.
• Optional and context-specific procedure blocks that may only need to be done in certain cases or done differently depending on something like the target environment.

What we will call a “Tracking Checklist” is used to perform the checklist without losing your place. It is where we track:

• The current real-time completion status of checklist items as the checklist is being processed.
• Who completed the checklist.
• When they completed it.
• What target they completed the checklist against (if there are multiple possibilities).
• Recording of variances compared to expected outputs or results.
• Whether variances were experienced that imply or directly indicate a template checklist update should be considered.

Template Checklists frequently need preparatory customization to be used as a Tracking Checklist. For example, the same template might be used to push changes to one of five environments because the process is identical except for a few bits of variable data, such as where the environment is located and the configuration details of that environment. These may be as simple as completing a table of details at the top, or may require more elaborate edits throughout. The preparation stage can also be important to the approval process — anyone whose pre-approval is required will want to see the specific details in a Prepared Tracking Checklist before giving the okay. Finally, Completed Tracking Checklists are an official record of the change event and generally need to be retained for audits and process-improvement studies.

The entire checklist lifecycle allows for continuous collaborative feedback against the Template Checklist. The value of a checklist is greatly enhanced as it aggregates more and more knowledge of what to do under various circumstances. If exception processes or infrequently executed sections get large, it may be necessary to break them out into their own sub-checklist template that it only invoked when needed.

Plain Markdown

If you are doing documentation of any type that is destined for Git, Markdown is, of course, the format of choice. The solution presented here focuses on flavorless, standard Markdown so as to be the most applicable to the most rendering environments as possible. The one small departure is the use of the SPAN or MARK html tag to add a color background to stamps when the rendering engine supports CSS. However, not only is this simply ignored in environments that don’t render CSS, the markings that use it also have standard markdown formatting to make them stand out sufficiently.

Tracking Checklist Tools and Storage

In Part 1 of this series, the concept of a “Master” checklist was discussed in passing. Due to the features of GitLab and GitHub, you may elect to treat storage of Template Checklists differently from Tracking Checklists.

Checklist Interactivity on GitLab and GitHub

The two major source-control systems support checkbox interactivity in specific item types of the system. These item types consist of Issues, Comments and Pull/Merge requests. Interactivity means you can directly click to check and uncheck checkboxes without having to first put the item in full “edit” mode. This makes checklist execution via the web site seem much more natural.

Since code should only be changed by a code review process, neither GitLab nor GitHub support checkbox interactivity in source code files.

This interactivity has a positive implication for real-time collaboration on checklist execution in that status updates to the checklist are realtime and do not require a commit-push cycle to update status. There is also no risk of merge conflicts if multiple individuals are updating the same checklist. Issues also have the advantage of tracking overall work visibility and completion and are regularly used for auditing completed work.

The interactivity allows non-coders to be a part of collaborative completion of checklists formatted in “markdown” — which is a standard markup language. This means they do not have to replicate repositories, configure editors, learn markdown syntax, learn how to commit and push code, etc. Even with the available online editors, users have to learn markdown and have to close out individual checkbox edits quickly if more than one user is processing or viewing a checklist at once.

When it comes to auditability of checkboxes in Issues, GitLab has a distinguishing feature in that it adds a comment to the issue discussion log that tracks what actions are taken on checkboxes (check or uncheck), who took the action and at what time the action occurred. This can be a massive help for general auditability. If checkboxes are carefully checked exactly when actions or wait states are completed, it can also help provide metrics for how long specific parts of the process take. Elapsed time information can help when attempting to optimize slow automation processes and/or slow human-performed steps. It is a further help on checklists that are executed by more than one person at a time as you can know who completed which steps.

Template Checklist Tools and Storage

Since changes to template checklists should most likely be reviewed by more than the individual making the changes, they should be stored directly in git so that changes can be subject to collaboration and review requirements the same way machine code is managed.

How you intend to keep completed checklist records may affect where you store the masters. If completed checklists are considered permanent records by remaining in completed issues and pull/merge requests, you should keep in mind they are not part of the git records of the project and won’t transfer if you move the git file system. If “historic records” are part of your GitOps discipline, then you may want at least a copy of the completed checklists in a source code folder. However, if the working checklists are stored in Git, they are not interactive on the Web, which makes them less functional — especially for real-time collaboration.

Here are some storage options to consider for completed tracking checklists (template checklists will always be somewhere in source):

1. Only in source code.
2. Only in completed Issues and pull/merge requests.
3. Both as an Issue or pull/merge request with a copy into git for record-keeping sake.

The third option of storing the checklist in both places allows easy discoverability and real-time collaboration on execution. It also allows you to be selective about which checklists become a permanent record — perhaps only “Promote to Production Checklist” needs to be kept rather than all the pre-prod environment checklists as well?

Template Storage in Github and Gitlab

The reason that completed checklist storage matters when considering storage of template checklists is that both GitLab and Github support placing markdown files in special locations in the git file system that make them into templates. This allows them to both be under collaborative source control, but also function as easy to discover templates for Issues and pull/merge requests in the web UI of each system. These are also the same subsystems that support interactive checklist usage. So you can imagine having template checklist templates like “Release Preparation Checklist”, “Promote to Staging Checklist” that are easily selectable when creating Issues or pull/merge requests and that require collaborative review in order to update.

Here are the special subfolder locations that enable markdown documents to be template — although some alternate locations are also possible. These specific locations in each product allow for multiple templates for each supported type and are also hidden when making local repository clones.

GitLab:

• .gitlab/issue_templates/
• .gitlab/merge_request_templates/

Github:

• .github/ISSUE_TEMPLATE/
• .github/PULL_REQUEST_TEMPLATE/

Pull/Merge Requests for Preparing Checklists

While GitLab and GitHub support checklist processing in both issues and pull/merge requests, both systems only support approvals on merger/pull requests. You may want to consider the merging of something benign, such as a token text file update, as a method of providing approvals for the checklist preparation. The resulting prepared checklist could then be inserted into an issue or a real merge request.

Pull/Merge Requests Approvals for Official Gating of Checklist Completion

GitHub and GitLab both support the concept of creating approval workflows for pull/merge requests, which enable checklist completion to be a much stronger control when the pull/merge request consists of a checklist that must be completed. Due to its embrace of GitOps, GitLab is notably more feature-rich in this area. Since GitLab also manages environment deployments, merge requests play a key role, not only in code review and signoff but also in deployment review and signoff. Multiple merge approval rules, electronic signatures, builds for merge commits and over ten other controls are available in various editions of GitLab.

GitHub Nuances

GitHub templates must have YAML front matter added to the beginning of the document in order to appear as a selectable template when creating issues. The following repositories contain examples of merge requests and issue templates in the above locations as well as in-progress checklists as an issue and merge request.

GitHub currently does not have a selector for multiple pull request templates, you must use http query parameters to select one even in the GUI.

GitLab only requires that the files be placed in the appropriate folder and then they will be available in the template selector in the web UI. This also means the user only has the title by which to initially judge the suitability of a template — file naming that clearly expresses and distinguishes the purpose of each template is necessary.

Creating Placeholder Tokens for Information Insertion

A great way to indicate the need to insert specific data in a checklist is to use a code fence with a unique, standardized token like this: _REPLACE_WITH_ (some text) This gives multiple advantages over your own custom solution to indicating where to record result information:

• Code fences are already supported with unique highlighting (usually inverse) in most markdown editors and rendering engines.
• They can be placed within the context of almost any other markdown (tables, bullets, blockquotes, etc).
• The use of a replacement token can be directly used by humans or an editor’s search and replace functionality or used by automation (like CopyQ).
• The replacement token method can be used for rapid checklist preparation of information that repeats throughout the checklist, such as searching and replacing all occurrences of ‘_REPLACE_WITH_ X.Y’ to the actual changeset version number in one or more checklists in one operation.
• In addition to checklist preparation, the replacement token method works equally well for values that will be completed during execution of the checklist, such as _REPLACE_WITH_ screenshot-of-messages-log.

Sample Repositories

These sample repositories include templates as well as in-progress issues and merge requests containing partially complete checklists to help you see how issue and pull/merge request templates work and how interactive checkboxes work:

https://gitlab.com/missionimpossiblecode/gitops-for-human-processed-checklists

https://github.com/darwinjs/GitOpsforHumanProcessedChecklists

Sometimes You Need Even More

This article and the sample repositories give a fairly comprehensive view of what is possible with the templates and interactive checklist support built into the most common git collaboration systems. Our final article in this series will look at multiplatform desktop utilities for a much richer markdown editing experience and the ability to insert standardized markdown formatted snippets during Template Checklist creation or Tracking Checklist completion.

Photo by Darlene Alderson: https://www.pexels.com/photo/person-sitting-on-sofa-using-laptop-7971328/

We are all familiar with the concept of a computer executing code — it takes a structured instructions and executes them in order to produce a desired end result. Computers are very good at this meticulous duty and perform instructions flawlessly many times over.

So what would the equivalent be in the human realm? How about a detailed set of ordered steps that must be completed to achieve a desired result? That seems like an apt description of procedural checklists.

The book “The Checklist Manifesto” paints a picture of how checklist usage creates a dramatic impact on quality outcomes by achieving consistency in meticulous and repetitive human-performed tasks. Checklists literally save lives in many high stakes circumstances like flying airplanes and performing surgery.

GitOps is a commitment to ensure everything needed for success in building and operating software is stored in Git. In fact, Git ultimately serves as the single source of truth. However, most of the time we secretly limit that “everything” to “everything the computer performs” and leave out the things that humans do to ensure the integrity of the software.

While a true commitment to the “everything in Git” principle would encourage the storage of checklists that humans perform in Git, there are some much deeper connections between the human code and the computer code that eventually automates these procedures.

The real focus here is operational checklists for the success of software (although we’ll discuss a lot about checklists in general). Think of them as build, deployment or testing steps.

The Genesis of All Process Oriented Work

Process-oriented computer work has its roots in work done by humans. We want to be relieved of the tedium of precise, ensured execution of extremely detailed, order-sensitive steps. If that work is done in a repeating cycle, we have all the more motivation to be rid of and to quality ensure it by giving the work to machines that don’t experience mental drift due to tedium.

The Best Process Discovery Mechanism

I have had periods in my career where overnight air travel was frequent. Since I have a busy mind, the possibility of forgetting to bring something was very real and caused a lot of stress. Overnight air travel involves several “point of no return” milestones because at a certain point you no longer have available time to double back to your home to grab an item and later, you can no longer double back to your vehicle to grab something either.

The power of a master checklist to solve this has been amazing. Over the years of keeping this checklist, I’ve noticed something else: that the process of creating and executing the checklist causes things to come to mind that need to be in it (or removed from it).

Sometimes these are completely new types of things that I hadn’t previously thought of putting on the checklist. This checklist is now ordered by T-minus transitions — the same way spacecraft launch checklists are tracked. Things like “Before Leaving the House” and “Last Minute Grabs From the Car.” It also has conditionals like If International Travel, If Empty House.

Brain Dumping and Mental Associations

It seems when we create a concrete checklist from our tacit knowledge of a process, it activates our minds to find related items. This is a natural part of human cognition, but is especially helpful with regard to discovering edge cases and conditional steps. The long term maintenance of a master checklist also continues to add value in this area as there is someplace to put new ideas about what ought to be done to make a checklist more effective at assuring its proper execution.

Tacit Process Knowledge Made Explicit

When we build software to take the place of human process-driven work, the human checklist is an invaluable source of information. Even if the checklist has to be made specifically for a software build, it provides a full extraction of the tacit knowledge of those performing the processes because it can be validated by someone with no tacit knowledge — which is exactly what computer code has to do because it does not have tacit knowledge either. Tacit process knowledge is the knowhow that experts have a hard time explaining to you because it has become so reflexive to them they do not notice that it is an important part of process execution for those who are unfamiliar.

Development Backlog Refinement

The building and refining of checklists really represents a somewhat automatic refinement of process automation requirements. Rather than trying to empirically discover what should be done while coding how to do it — discovery and coding can be separated. This helps when there are different individuals doing each activity, but it also helps when the same individual does the activity because separating the flow states of “coding” and of “focused checklist execution” yields the top-notch awareness that improves the quality of outcomes for both activities.

Discerning Low and Negative Automation ROI

While the “Automate All The Things” meme is funny, it is also a commentary on a philosophical position taken too far. Automation is about efficiency, which in turn is about return on investment. Several brave writers and bloggers have published guidelines on situations where the ROI is not there for automation. These tend to be the areas where human judgment and/or strong variability between executions are an inherent characteristic of a process. Checklists help discern where automation ROI may be low because the attempt to create a checklist with a lot of judgment calls helps articulate whether the code required to make the same judgments is possible and cost-effective. Another indicator is that a checklist needs new custom steps for every execution. These characteristics indicate there would be more time spent maintaining the code of the automation than is involved in humans performing the tasks — perhaps by orders of magnitude. The process of attempting to automate can also weed out which sub-processes which naturally have a low automation ROI, while also automating anything that is a candidate for automation. Discerning low or negative ROI can be done during the overall automation discovery process — but the existence of low/negative ROI must be believed in and filtered for to avoid the “automate it no matter the ROI” trap.

Human Code Update, Review and Adoption

Just as computer code benefits from collaborative-maintenance updates, so human code — in the form of master checklists — can also benefit from this collaborative approach — whether or not it can ever be boiled down to automation or not. Collaborative coding is a main feature of Git and gaining the benefit is as simple as ensuring checklists are stored in Git, in a text format so that Git can perform human-readable diffs on changes. Essentially this naturally leads choosing Markdown as the document format.

Human Code Auditability

Since humans have a challenge performing meticulous, repetitive activities with high fidelity, we tend to want them to provide proof that they performed the actions as prescribed or describe what alternative action was taken. By storing completed checklists in Git, we can facilitate auditability. As discussed later, we can determine what type of “DONE” marking is used in order to improve the auditability — such as whether to include a date time stamp in the DONE marking of each step.

Automation and Checklists Go Better Together

Some DevOps Engineers might experience frustration over spending effort on codifying checklists rather than directly creating computer code to do those activities. The above list shows very valid reasons why it makes sense to acknowledge and purposely incorporate checklists into the process of initial and ongoing maintenance of automation. The risk that something might not be automated because it is too well documented as a human procedure should not lead us to ignore the fact that non-codified human procedures are a large scale risk to the integrity of the full DevOps lifecycle of a software solution.

Live Checklist Status Sharing

During a change event, it is very handy to understand where everything is in the process — especially if multiple checklists must be performed simultaneously. While some of the SaaS checklist solutions offer live sharing as a primary value, Git-stored lists can come close if the checklist being watched is frequently pushed to a git server and individuals viewing it refresh their view. You could even set up a looping “git commit and push” script on the workstation executing the checklist.

Code Is Code Is Code and Should Be Stored in Git

It’s common to think of “code” as being computer code — but human code fits the model in the form of checklists, with very detailed, order-specific and interdependent execution instructions. When we record human code in a way that leads to effective execution — we naturally arrive at the concept of a checklist. As we’ve discussed human code has many similarities to computer code — including how it benefits from the visibility and collaborative capabilities Git offers.

When human code is critical to the development, build, testing and operation of a software stack — it fits the definitions of GitOps — the single source of truth for success in managing a software stack.

Well-written and -maintained human code is the best starting point for creating computer code.

In a follow-up post we will look at two free, open source and multiplatform tools that make the creation and execution of markdown-based checklists much easier. It will also include configuration code that gets you started with easy “done marking” to keep the stress of checklist execution as low as possible and consistent between team members.

Early on it was evident that for OS and software provisioning it was extremely important to be able to prove that you had documented the exact checklist of ordered steps to take a system from “pristine OS deployment” to “working configuration” as an input to solid automation code.

Over time, I developed proficiency in this type of reverse engineering, explored the hundreds of available free and commercial tools, blogged about it and eventually developed a business around multiple advanced training courses on reverse engineering Windows OS and software installation for the purposes of provisioning and deployment automation.

There is an entire eco system of commercial and free tools for system-wide diffing on Windows, so it was a bit shocking to recently rediscover that comprehensive system-wide configuration diffing tools for Linux are rare indeed.

Whether you are porting Windows reverse engineering skills to Linux, or are a Linux Engineer that has not experienced the productivity benefits of system-wide config diffing for reverse engineering - this post is for you ;)

Reversing a Working Raspian Configuration

For a robotics control project, I was recently setting up Raspian (the Raspberry Pi distro) to enable Bluetooth audio access via a running service. Bluetooth audio on Raspian needed a lot of work just to be functional. The requirement to simultaneously using the GPIO serial port to control an ardruino based robot created some knock-on complexity in figuring things out. Linux Bluetooth audio is also very per-user oriented (e.g. uses a user based systemd service and config files) because Bluetooth audio devices are frequently personal to users. However, it needed to work from the context of a systemd service with a special user logon so that the Bluetooth speaker could appear to give the robot commands.

The end result worked great - so I began the work to build “from scratch” instructions so others could do the same. However, the path to getting it working was extremely unclear due to the many changes (including unrelated ones) made along the way.

The last few steps to get Bluetooth audio working under a service remained unclear no matter how many things I tried or compared or repeated.

Devising a System-Wide Linux Config Diffing Solution

I mistakenly thought there would be a couple obvious solutions available to do a system-wide diff to quickly isolate the differences between the working and from scratch build.

I was unable to find any obvious options, but eventually found configsnap - created by the venerable Rackspace support team.

While it looked very promising, I hit additional snags - the help and documentatoin did not cover whether the concept of comparing two different machines was a valid use case and the repository docs did not show compare commands at all. Searching the Internet did not yield any how to documents or videos.

Whenever there is a gap this large (lack of linux system-wide diffing solutions in general and lack of how to information for the one I found) I can’t help but create a bit of Mission Impossible Code to bridge the gap.

The Mission Objectives and Parameters

 Mission Objectives and Parameters articulate the final objectives that emerged from both the preplanning and build process. Code Summary gives an out line of the code fragments. Code Call Outs highlights significant constraints, innovations and possible alternatives in the code.

1. Objective: Use tooling to find system-wide configuration differences between two seperate linux installations to quickly isolate the differences between a known good and non-working system.
2. Desirable Constraints In Meeting Objective:
   1. Origination Priorities:
      1. Source a ready-made solution, but if that fails…
      2. Assemble a solution from existing bits and pieces, but I don’t have the time to
      3. Build a solution from scratch.
   2. Leverage a “Zero Footprint” methodology where the use of the diffing tool does not create substantial configuration changes.
   3. Work on as many distros as possible (Configsnap is packaged for only some distros)

Code Summary

1. Using a Zero Footprint approach, use minimal code to bring down configsnap and run a before snapshot.

   1. If possible, enable “run from web”.

2. Perform a “known good” snapshot on the working reference system.

   1. Use a custom snapshot name that self-identifies its purpose (“crossmachinecompare”)
   2. Use a custom stage name that self-identifies its purpose (“knowngoodconfig”)
3. Provide sample commands for direct compare on the “compare-to” system.

Code Call Outs

Running Directly from Web

• The provided commands works around the fact that some systems block a directly downloaded script from being piped directly into bash - while a bit longer, this command works on a broader array of linux machines.

Zero Footprint

• Zero Footprint is a constraint that avoids changing any system-wide configuration to use code on a target system. Generally everything runs out of a directory and package managers are not used. This is more important for diffing utilities since they should not place resources that end up in the diff itself - especially if they are assessing production systems. In the past, I have had to create a zero footprint install of the CIS tool used to assess CIS Benchmark Hardening for both Windows and Linux.

Provided System-Wide ‘additional.conf’ Example

• By default the tool does not compare all of /etc/ nor any user configuration files. The example shows the most basic level of including these important configuration areas
• The example configuration provides an easy to extend example for refining the scope of comparison.

Complete Single Script Solution

• By creating it’s own configuration file and emitting the commands to use on “compare-to” target systems, the single script is fully self-contained and self-documenting.
• Single script solutions are frequently easier to automate since many management systems allow embedded transport of scripts (but not of packages, support binaries, etc).

The Code Itself

#Run this directly from this location with: curl https://gitlab.com/missionimpossiblecode/MissionImpossibleCode/-/raw/master/ConfigsnapCreateKnownGood.sh -O /tmp/ConfigsnapCreateKnownGood.sh ; sudo bash /tmp/ConfigsnapCreateKnownGood.sh

#Zerofootprint for both known good and compare-to systems - just delete /tmp/configsnap

if [[ -z "$(command -v python)" ]]; then 
  echo "Python must be installed and working, exiting..."
  echo "If you cannot install python on this or the compare-to system, read here about building it in an isolated directory: https://stackoverflow.com/a/42903156"
  exit 5
fi
mkdir -p /tmp/configsnap
curl https://raw.githubusercontent.com/rackerlabs/configsnap/master/configsnap -o /tmp/configsnap/configsnap
chmod +x /tmp/configsnap/configsnap

cat > /tmp/configsnap/additional.conf <<'EOF_CONFIG'
[allmachineconfig]
Type: directory
Directory: /etc/

[userconfigs]
Type: directory
Directory: /home/
File_Pattern: \..*
EOF_CONFIG

sudo ./configsnap --basedir=/tmp/configsnap/snaps --verbose --tag=crossmachinecompare --phase=knowngoodconfig

cat <<- EndOfMessage

Next Steps:

1. Sample scp command to pull this on a system to compare to: 
   scp -r user_on_this_system@thissystemdnsorip:/tmp/configsnap /tmp/configsnap
2. Sample auto-compare command on compare-to system:
   sudo /tmp/configsnap/configsnap --basedir=/tmp/configsnap/snaps --verbose --tag=crossmachinecompare --pre=knowngoodconfig --phase=post

To use as a known good snapshot managed in a centralized location, copy "/tmp/configsnap" to a shared location (or use git to commit to a repository) where you can pull it onto any system you wish to test for drift or changes.

To clean the zero footprint install from any systems, run "sudo rm -rf /tmp/configsnap"

EndOfMessage

Source Code for This Post

The code for this post is kept up to date and can be invoked directly from the web in this repository location: ConfigsnapCreateKnownGood.sh

Mission Impossible Code Series Inclusion

• The solution sticks to the Boring Technology selection criteria.
• The solution is implemented in a single script.
• The solution is Zero Footprint.
• The solution is portable between linux distros and comparison systems.

Solution Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

 The following content is a deep dive below the waterline into the nitty gritty details of how to take a similar approach to building solutions.

NOTE: You do not need this information to successfully leverage this solution.

What Does “<==>” Mean?

The notation “<==>”, which may contain logic like “<= AND =>” is an attempt to visually reflect the trade-offs inherent in using heuristics to commit to seleting a position on a spectrum of possibilities. By documenting these trade-offs below - the construction and serendipities of the final tuning are revealed. This seems to do at least three things for the consumer of this information:

1. You get to see the iceberg below the waterline of something I have built that I hope is “As simple as possible, but not simpler.” So you get to see why I claim that “The Creation of Simplicity is Necessarily a Complex Undertaking.”
2. You can more easily customize key parts of the solution to your liking, and not suffer from unintended consequences of those changes.
3. You can more easily apply this pattern to new problems that may be similar, but not identical.

Solution Architecture Heuristics

The overall solution is solving for “Use tooling to find system-wide configuration differences between two seperate linux installations to quickly isolate the differences between a known good and non-working system.”

Requirement: (Satisfied) Be Self Contained (Including Instructions)

• Mission Impossible Heuristic: Bring Everything You Depend On <= AND => Pack Light (Reference System)
• Reason: The more local dependencies a solution requires, the less portable it is and the more challenging it is to reuse across varying configurations. The gold standard is if the script can be used without instructions via the code containing embedded configuration and embedded instructions as needed.
• Coding Decisions:
  • Bring Everything: Use curl to download a raw copy of the python code from the repository - thereby avoiding dependencies on package managers (only a RHEL package exists) and Git (in the case of cloning the entire repo).
  • Bring Everything: Use configuration as code via a heredoc that creates the configuration file. This enables the entire solution to be in a single, run-from-web script. Configuration as code is also self-documenting by nature which avoids the need for external instructions.
  • Bring Everything: The Reference System code emits the instructions to be used on the Target Systems - providing User Instructions as Code also enables the self-contained, self-documenting nature of the solution.
  • Pack Light: While some few linux images (esp containers) may not include python, if it cannot be found, ask the user to resolve the dependency - with a hint on how to build python from source and run it without updating the entire system with Python.

Desirement: (Satisfied) Zero Footprint Approach

• Mission Impossible Heuristic: Leave No Trace Behind <= AND => Make Fingerprint Wipe-Down Easy

• Reasons:

  • Dependencies and system level configuration require permissions and soil the system. Drift detection on hardened systems (e.g. CIS Benchmark) or change management regulated systems (e.g. FDA Regulated) generally should not have their system level configuration or files changed .
  • The diffing tool itself must ensure that it’s own code, config and data do not become part of the comparison it is performing.

• Coding Decisions:

  • Use directory tree in /tmp.
  • Store Code, Config (additional.conf) and Snapshot Data in the same directory tree.
  • Cleanup is as easy as removing the root of the directory tree containing the code, config and data.

Serendipity: (Discovered) : Bring Everything You Depend On <= AND => Pack Light (Target System(s))

• Prior Requirement / Desirement: Be Self Contained, Zero Footprint Approach
• Prior Coding Decisions Result:
  • By storing Code, Config and Snapshot Data in the same directory heirarchy the resultant directory can be copied to a central location or repository or directly to a Target System and it is immediately runable with only two commands (even less prep than the Reference System)

Serendipity: (Discovered) Least Privilege Approach

• Prior Requirement / Desirement: Zero Footprint Approach
• Prior Coding Decisions Result:
  • By using /tmp and storing Code, Config and Snapshot Data, special permissions are only needed to run ‘Configsnap’ itself and then the operations of Configsnap only change a temporary, non-tracked area of the system.

Serendipity: (Discovered) Identical Configsnap Version, Configuration and Baseline Snapshot

• Diffing Heuristic: Two comparison targets should not have captured differences introduced by the comparison process.

• Reason: When diffing across two systems, the version of the diffing utility (Configsnap) and it’s configuration must be identical for results to be valid. This includes using older versions of the diffing utility even if they are no longer available from the original source.

• Prior Coding Decisions Result:

  • By storing Code, Config and Snapshot Data in one directory tree the version of all these components is frozen at the time point when the “Reference Snapshot” was taken.

Requirement: (Satisfied) Maximize Applicable Linux Systems This Can Be Used With

• Mission Impossible Heuristic: Optimize Your Choices <= AND => DeOptimize To Match The Breadth Of Required Scope
• Reasons: The usefulness of system-wide snapshots is applicable to all linux distros and architectures, ensure the proposed solutions reaches for this same scope.
• Coding Decisions:
  • Use curl to download a raw copy of the python code from the repository - thereby avoiding dependencies on package managers. Package managers complicate things because
    • The utility must have already been packaged for that package manager platform.
    • The utility must have a package per OS architecture (e.g. x86_64 and arm)
    • The package preparation must occur frequently enough to have the latest version of the software.
    • There are many different script commands to accomodate all possible package managers.

"The Whole Berg - Above and Below the Waterline" Posts in the “Mission Impossible Code” Series contain toolsmithing information that is not necessary to reuse the solution - use the iceberg glyphs to know when the content is diving below the water line into “How I Made This”. The content is also designed to be skim-read.

"Tip of the Iceberg - Concise Summary Discussion" The “Tip of the Iceberg” icon indicates as simple as possible info on why and what in order to assess and implement.

"Deep Dive - Below The Water Line Discussion" The “Below The Water Line” icon indicates a deep dive into nitty gritty details of how to take a similar approach to building solutions.

Pearl Diving To Learn Custom Resources, Lambda and Python

During this effort I mused how we sometimes do the equivalent of Pearl Diving when taking on new skills. Its the idea of deep learning a stack of one or more new things while under the pressure of needing the end state code to reflect a maturity level significantly higher than your beginner expertise in that stack. I have captured the details about Perl Diving - and when you should use it (because you usually should not) - in a companion post titled Pearl Diving - Just In Time Learning of Mature Coding Habits For a New Stack

The Mission Objectives and Parameters

"Tip of the Iceberg - Concise Summary Discussion" Mission Objectives and Parameters articulate the final objectives that emerged from both the preplanning and build process. Code Summary gives an out line of the code fragments. Code Call Outs highlights significant constraints, innovations and possible alternatives in the code.

1. Objective: Ensure that AWS VPC / Networking can be specified, with minimal complication of the existing easiest user experience case
2. Desirable Constraints In Meeting Objective:
   1. Adding the minimum number of new parameters to give the ability to select a network location for the scaling group.
   2. Retaining simplicity of previous version to automatically use the default VPC by default.
   3. Do not exceed code size limitations that would complicate the solution beyond the current simplicity of a single CloudFormation template (just for the sake of code size). Limit for CloudFormation embedded Lambda functions: 4KB . Limit for CloudFormation template file size when loaded from S3: 460.8 KB (note: the limit when submitting from the command line is 51.2 KB)

Code Summary

1. Shows minimal CloudFormation (CF) Custom Resource, consisting of 3 CloudFormation Resources
   1. The Custom Function declaration - which contains the Python code (“VPCInfoLambda:” in the below)
   2. The IAM Role declaration - permissions used by the Lambda execution (“CFCustomResourceLambdaRole:” in the below)
   3. The “call” to the Custom Function - the input of parameters and return of data through a Cloud Formation Resource interface (“LookupVPCInfo:” in the below)
2. A fragment shows how the resource data is retrieved inside the definition of a scaling group. (“InstanceASG:” in the below)
3. A fragment shows the definition of the parameter (“Parameters:” in the below)

Code Call Outs

Using The Code For Subnet Enumeration

• Note that the parameter “SpecifyVPCToUse” does not use the type “AWS::EC2::VPC::Id” to get a drop down list of actual VPCs because it would defeat the above Desirable Constraint “Retaining simplicity of previous version to automatically use the default VPC by default.” However, if your organization NEVER uses default VPCs or disables them, changing the parameter type to “AWS::EC2::VPC::Id” actually improves the experience because users do not have to lookup VPC ids in the console and the one they select will always exist.
• Note that the CF functions that retrieve data for “AvailabilityZones:” and “VPCZoneIdentifier:” could add the !Select function to only use a specified number of AZs and Subnets rather than using all available Subnets in the VPC.

Reusing The Code As The Pattern For Other Custom Resources

• Note that Lambda logging to CloudFormation is configured (including security) and helpful account and region context are output whenever trapped or untrapped exceptions occur.
• Note the building of the object “responseData” - this is showing how to return multiple values when many examples show a much simpler structure for returning only a single value.
• Note lines with “raise Exception” reuse the global exception handling for trapped known exceptions to keep code compact by reusing the logging, tracing and verbose error output code containing context cues.
• Note the lines with “signal.alarm” are timeout handling - which is important in serverless.
• Note that “CFCustomResourceLambdaRole:” is the least privilege IAM permissions for this function. If you build a function to do other things, the permissions in this YAML will need to be updated to match - but should be kept least privilege.

Source Code For This Article

This Code Working In Production

This code was created for the solution GitLab HA Scaling Runner Vending Machine for AWS

The Code Itself

#Arguments: Vpc-id or "DefaultVPC"
#Returns: vpc-id, number of subnets and ordered list of subnetids and az ids.  
# The index of these two return lists are correlated if it is desirable to choose less than the whole list using the CloudFormation function "Select" against both lists.
  VPCInfoLambda:
    Type: 'AWS::Lambda::Function'
    Properties:
      Description: Returns the lowercase version of a string
      MemorySize: 256
      Runtime: python3.8
      Handler: index.handler
      Role: !GetAtt CFCustomResourceLambdaRole.Arn
      Timeout: 240
      Code:
        ZipFile: |
          import logging
          import traceback
          import signal
          import cfnresponse
          import boto3

          LOGGER = logging.getLogger()
          LOGGER.setLevel(logging.INFO)

          def handler(event, context):
              # Setup alarm for remaining runtime minus a second
              signal.alarm((int(context.get_remaining_time_in_millis() / 1000)) - 1)
              try:
                  LOGGER.info('REQUEST RECEIVED:\n %s', event)
                  LOGGER.info('REQUEST RECEIVED:\n %s', context)
                  if event['RequestType'] == 'Delete':
                      LOGGER.info('DELETE!')
                      cfnresponse.send(event, context, "SUCCESS", {
                           "Message": "Resource deletion successful!"})
                      return
                  elif event['RequestType'] == 'Update':
                      LOGGER.info('UPDATE!')
                      cfnresponse.send(event, context, "SUCCESS",{
                           "Message": "Resource update successful!"})
                  elif event['RequestType'] == 'Create':
                      LOGGER.info('CREATE!')
                      request_properties = event.get('ResourceProperties', None)

                      VpcToGet = event['ResourceProperties'].get('VpcToGet', '')
                      ec2 = boto3.resource('ec2')
                      VpcCheckedList = []
                      TargetVPC = None
                      vpclist = ec2.vpcs.all()
                      for vpc in vpclist:
                          VpcCheckedList.append(vpc.id)
                          if VpcToGet == "DefaultVPC" and vpc.is_default == True:
                              TargetVPC=vpc
                          elif vpc.vpc_id == VpcToGet:
                              TargetVPC=vpc

                      if TargetVPC == None:
                        raise Exception(f'VPC {VpcToGet} was not found among the ones in this account and region, VPC which are: {", ".join(VpcCheckedList)}')
                      else:
                        VPCOutput = TargetVPC.id
                        subidlist = []
                        zoneidlist = []
                        subnets = list(TargetVPC.subnets.all())
                        for subnet in subnets:
                          subidlist.append(subnet.id)
                          zoneidlist.append(subnet.availability_zone)
                        subidOutput = ",".join(subidlist)
                        zoneidOutput = ",".join(zoneidlist)
                        if not subnets:
                          raise Exception(f'There are no subnets in VPC: {VpcToGet}')
                        LOGGER.info('subnet ids are: %s', subidOutput)
                        LOGGER.info('zone ids are: %s', zoneidOutput)

                      responseData = {}
                      responseData['VPC_id'] = VPCOutput
                      responseData['OrderedSubnetIdList'] = subidOutput
                      responseData['OrderedZoneIdList'] = zoneidOutput
                      responseData['SubnetCount'] = len(subidlist)
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)                                  

              except Exception as err:
                  AccountRegionInfo=f'Occured in Account {context.invoked_function_arn.split(":")[4]} in region {context.invoked_function_arn.split(":")[3]}'
                  FinalMsg=str(err) + ' ' + AccountRegionInfo
                  LOGGER.info('ERROR: %s', FinalMsg)
                  LOGGER.info('TRACEBACK %s', traceback.print_tb(err.__traceback__))
                  cfnresponse.send(event, context, "FAILED", {
                      "Message": "{FinalMsg}"})

          def timeout_handler(_signal, _frame):
              '''Handle SIGALRM'''
              raise Exception('Time exceeded')

          signal.signal(signal.SIGALRM, timeout_handler)


  #Custom Function IAM Role Declaration
  CFCustomResourceLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        - PolicyName: "lambda-write-logs"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"  
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*"
        - PolicyName: "describe-vpcs-and-subnets"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "ec2:DescribeVpcs"
                  - "ec2:DescribeSubnets"
                Resource: "*"

  #Calling Function to Retrieve Data
  LookupVPCInfo:
    Type: Custom::VPCInfo
    Properties:
      ServiceToken: !GetAtt VPCInfoLambda.Arn
      VpcToGet: !Ref SpecifyVPCToUse

Fragment That Demonstrates Parameter Collection

#Parameter declaration with important default
Parameters:
  SpecifyVPCToUse:
    Description: >
      DefaultVPC - finds the VPC and configures all of its subnets for you. Otherwise type 
      in the VPC id of a VPC in the same region where you run the template. 
      All subnets and azs of the chosen vpc will be used.
      The VPC and chosen subnets must be setup in a way that allows the runner instances 
      to resolve the DNS name and connect to port 443 on the GitLab instance URL you provide.      
    Default: DefaultVPC
    Type: String
    # While it is tempting to make the above parameter of type "AWS::EC2::VPC::Id" 
    # this prevents automatic discovery and usage of the DefaultVPC.
    # However, if your organization NEVER uses default VPCs or disables them, changing 
    # the type to AWS::EC2::VPC::Id actually improves the user experience because users do not have to 
    # lookup VPC ids in the console.

#Fragment showing using the resultant data from the custom function
  InstanceASG:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AvailabilityZones: !Split [",",!GetAtt LookupVPCInfo.OrderedZoneIdList]
      VPCZoneIdentifier: !Split [",",!GetAtt LookupVPCInfo.OrderedSubnetIdList]

Solution Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

"Deep Dive - Below The Water Line Discussion" The following content is a deep dive below the waterline into the nitty gritty details of how to take a similar approach to building solutions.

NOTE: You do not need this information to successfully leverage this solution.

The following list demonstrates the Architectural thrust of the solution. This approach is intended to be pure to simplicity of operation and maintenance, rather than purity of a language or framework or development methodology. It is also intended to have the least possible dependencies. The below is a mix of a) previously committed dispositions for the Overall Solution, b) predetermined design points and c) things discovered and adopted during the development process (emergent or organic solution architecture component).

What Does “<==>” Mean?

The notation “<==>”, which may contain logic like “<= AND =>” is my attempt to visually reflect the dynamic tension or trade-offs inherent in using heuristics to commit to fixing positions on a spectrum of possibilities. During the solution formulation these positions fluctuate as you try to simultaneously tune multiple, interacting vectors through trial and error. Even when I do it on purpose, I still can’t completely understand how I am tuning multiple vectors at once and why the results of the process repetitively turn out to effectively solve for multiple vectors. However the internals work, once you’ve produced a sufficiently satisfactory solution, their final positions reflect a complete tuning. They are sort of like custom presets on a sound equalizer. By documenting them as I have done here - I reveal my impression of the final tuning. I feel this does at least three things for the consumer of this information:

1. You get to see the iceberg below the waterline of something I have built that I hope is “As simple as possible, but not simpler.” So you get to see why I claim that “The Creation of Simplicity is Necessarily a Complex Undertaking.”
2. You can more easily customize key parts of the solution to your liking, while retaining the value attached to other parts of the tuning.
3. You can more easily apply this pattern to new problems that may be like it, but not identical.

Solution Architecture Heuristics for the Overall Solution

The overall solution is solving for “Allow Network Configuration Selection, Using the Least New Parameters and Without Complicating the Existing Easiest User Experience Case”

• Overall Solution Requirement: (Satisfied) Ensure that VPC / networking can be specified.

  • Benefits: Accommodate advanced AWS implementations generally have a design to their VPCs and may even disable the Default VPC. Most CloudFormation templates that are generalized to being a tool must provide this flexibility.
    • Coding Decisions: To actually make the VPC selection enhancement.

• Overall Solution Requirement: (Satisfied) Add the minimum number of new parameters to give the ability to select a network location for the scaling group.

  • Mission Impossible Heuristic: Make Sophisticated Spy Gadgets <=AND=> Have Simple Controls
    • Benefits: Adoption Driven Development: simple understanding of parameters, simple information collection, easy adoption of new version.
    • Discarded: Innovation Over Defacto Alternatives: It is very common for these types of templates to expose two parameters to take a list of subnet IDs and a list of availability zone names. It is then incumbent on the user to collect these two lists of IDs, ensure they match the desired VPC and ensure that the zones list exactly correlates to the zones of the selected subnets.
    • Coding Decisions: One parameter for the target VPC is used and defaults to the special value “DefaultVPC”. Users who need to use a specific VPC are more likely to know how to look it up or have been provided it by an Infrastructure Automation engineer. Another possibility is that experienced Infrastructure Automation engineer’s create code over this that forces specific VPCs to be used.

• Overall Solution Requirement: (Satisfied) Retaining simplicity of previous version to automatically use the default VPC by default.

  • Mission Impossible Heuristic: Make Sophisticated Spy Gadgets <=AND=> Have Simple Controls
    • Benefits: the solution is much simpler to use for beginners or when simply kicking the tires.
    • Coding Decisions: The VPC parameter defaults to “DefaultVPC” - when this value is detected, it automatically locates the default VPC and enumerates it’s subnets and availability zones. This actually took a lot design and coding - compared to the previous very simple implementation which used a built-in CloudFormation function to lookup AZs.

• Overall Solution Limitation: Cannot specify subnets / availability zones

  • Reason: the solution only allows selecting a VPC and then it uses all subnets of the VPC - this is for the sake of simplicity.
  • Limitation Rationale: For a first MVC, being able to select VPC gives most of the benefits of enabling a user to use custom VPCs they have prepared according to their practices with a single parameter.
  • Limitation Architecture: Sometimes subnet level selection is provided because a region only has two AZs - this problem is avoided by enumerating existing subnets. Sometimes subnet level selection is provided because certain instance types don’t exist or are constantly exhausted in an AZ - this problem is avoided because the underlying template is capable of selecting multiple instance types, the unavailability of an instance type in an AZ can be managed by providing multiple instance types in the list.
  • Limitation Removal Anticipation: This solution was also engineered specifically to anticipate the removal of this limitation, yet with minimum parameters, by customizers or in a future release. The CloudFormation Custom Resource returns the total subnets found and “order guaranteed” lists of the subnets and availability zones. This would allow adding a single parameter “Number of Availability Zones To Use” - the !Select CF function could then only use that number of randomly selected subnets, yet with matching AZs in the AZ parameter to AutoScaling Groups.

• Overall Solution Limitation: Cannot use AWS::EC2::VPC::Id to make VPCs list a drop down in UI based template execution.

  • Reason: This parameter type cannot indicate nor default to the Default VPC. This would inordinately complicate the simplest user experience case which must be retained.

Solution Architecture Heuristics for the CloudFormation Custom Resource Working Pattern

CF Custom Resource Requirement: (Satisfied) Be compact.

• Mission Impossible Heuristic: Bring Everything You Depend On <= AND => Pack Light.
• Reason: Code and file size limitations above
• Coding Decisions: Reuse of the default exception handling for both unanticipated errors and for calling from trapped errors. Used boto3 “resources” over “clients” as code is less verbose and easier to read.

CF Custom Resource Requirement: (Satisfied) Implement proper exception handling <==> despite size concerns <==> be compact.

• Mission Impossible Heuristic: If You Are Going To Die, Write a Note About Who Done It and Where.
• Reasons: a) The call sack is deep and remote, b) debugging cycles are long for IaC
• Coding Decisions: Find out the most concise, functional Python exception handling (Pearl Dive) which is likely NOT the best practice exception handling. Call that exception handling for reporting of caught exceptions. This was implemented with Python exception handling and the Python traceback Python module.

CF Custom Resource Requirement: (Satisfied) Have exceptions report maximum context and, where possible, troubleshooting hints.

• Mission Impossible Heuristic: If You Are Going To Die, Write a Note About Who Done It and Where.
• Reasons: a) Improve troubleshooting of external problems (like a mismatch between account and/or region and a specified VPC). b) Like myself, other users of this function may not be experts in CF Custom Resources, Lambda or Python - they may be Deep Diving when trying to troubleshoot problems, c) To get more detail in Lambda logs for an error that was blocking progress - multiple blocking error conditions were immediately cleared upon implementing this.
• Coding Decisions: Extract the AWS Account ID and AWS Region from the Lambda execution context and report it in exceptions. When reporting a VPC Not Found type error, report the VPC list that was enumerated to give evidence that the VPC actually does not exist and a context hint because the enumerated VPCs can be found. Use the logging module.

CF Custom Resource Requirement: (Satisfied) Always build using least privileged security.

• Mission Impossible Heuristic: Disclose Everything Needed For Success <=AND=> Use a Need to Know Basis
• Reasons: a) It’s the right thing to do in all coding, but even more so when promoting or reusing patterns - bad sample code is a known attack vector, b) having “least privileged” security built-in fuels adoption and reuse
• Coding Decisions: Aside from standard CloudWatch logging access for Lambda, this function needed to read VPC data and enumerate subnets - so a new, clearly named, IAM policy was created and contained only “ec2:DescribeVpcs” and “ec2:DescribeSubnets”.

CF Custom Resource Desirement: (Satisfied) Leverage common and available Python modules to simplify the code.

• Mission Impossible Heuristic: Build The Best Tools <=AND=> Use Available Tools
• Reasons: Code is simpler to read and reuse and more compact.
• Coding Decisions: Use of modules: logging, traceback and signal. Use of module “cfnresponse” - many examples had a lot more code in order to manually implement a cloud formation response to the calling stack.

CF Custom Resource Desirement: (Satisfied) Have this function be the basis of a template for future reuse.

• Mission Impossible Heuristic: Use The Best Implementation <=AND=> Use Available Tools
• Reasons: I just always try to do this because real world problems are the best source of raw matter for working examples.
• Coding Decisions: Demonstrate many required capabilities such as: passing parameters in, passing multiple data values out, timeout handling, exception handling, use generic IAM CF block for future reuse.

CF Custom Resource Serendipity: (Satisfied) Support timeout functionality for Lambda serverless.

• Mission Impossible Heuristic: Use Ad-Hoc Innovation <=AND=> Use Existing Patterns
• Reasons: Eliminate the source of tough problems by leveraging the wisdom of examples.
• Coding Decisions: While looking through many coding examples I noticed a few were monitoring for a timeout and implemented it. It turned out that I had situations where I exceeded the timeout which informed tuning the maximum allowed run time. I also found out later from someone experienced in CF custom resources that the timeout is a critical bit of code. Most example code was missing this.

During the effort described in Building a Best Practice CloudFormation Custom Resource Pattern, it was necessary to learn and create mature-ish code in the following stack: CloudFormation Custom Resources, Lambda (to back the custom resource), Python and Boto3.

Pearl Diving is also one of the Mission Impossible Code Diametric Spectrums: Have Seasoned Skills Through Experience and Practice <=AND=> Do Pearl Diving To Learn Skills Deeply and Quickly.

Tim Poffenbarger and I were discussing what more universal human learning experience is similar to Pearl Diving that might help with understanding the idea. We came up with learning to drive a car. You have to simultaneously become fairly well skilled in: 1) operating the vehicle, 2) following unstated rules of the road, as well as 3) posted signs and 4) watch for the dynamic hazards posed by other vehicles. If you learned on a stick shift you also had a mini-stack inside of operating the vehicle as you got used 1a) easing the clutch, 1b) while easing the throttle and 1c) knowing when to shift. Should I even mention adding the emergency / parking break in order to do a uphill start in a standard? In any case, learning to drive seems to be another required Pearl Diving learning expedition that many of us go through - and then get to go through again when teaching our young ones to drive ;)

Pearl Diving is, in my opinion, usually a learning antipattern because it combines learning new languages and their best practices right away. So it is a place that I feel reluctance to go. To that end, I have documented some of the inherent problem / solution heuristics that forced the Pearl Diving path on this project.

The pressure of needing it to be fairly mature from the start included:

• Small Code Required: A prime objective of this solution is to avoid breaking any size barriers that would force it to be more than one CloudFormation template. Those limits are 4KB for a Lambda function (when inline in a CloudFormation template) and the overall CloudFormation template size limit of 460KB (when sourced from S3). To leave room for future enhancements to the overall solution, a persistent “size minimizing” rule must be applied. To this end, some finer best practices for Python coding were regressed to keep the code small.
• Deep Stack Debugging: Another reason for Pearl Diving mode on this project was the call stack complexity required for final testing. It happens within Python, which is within Lambda, which is within CloudFormation. So this forced the issue of needing a mature implementation of exception handling and logging - because it is the only way to debug that deep. This investment paid off massively as soon as it was made. A previous post discussing this common problem described it as “5 miles out in a tunnel at the bottom of a mine shaft”: At the Coal Face: Code for Debugging Deep PowerShell Execution
• IaC Test Cycles Are Long: Personally I feel that Infrastructure as Code iteration cycles can be quite long compared to other types of testing and prototyping. Combined with how deep CloudFormation Custom Resources are in the call stack, this increases the desire to invest in a working pattern for future functions - especially with exception handling and logging for debugging.
• High Reuse Potential: The specific need for a CloudFormation template consumer to select their networking location is very recurrent and many solutions are overly complex in what they require the end user to provide for parameters. This sub-solution will likely be reused many times.
• Near-Future Feature Request Probability: The current solution allows one to pick the VPC, but then it uses all subnets of that VPC. I am pretty sure I will receive a feature request for either a) selecting specific subnets or b) selecting less than all the subnets of the target VPC. To these ends the implementation returns ordered lists of subnets and AZs so they can be predictably correlated and it returns the total subnets (index) - both of these are unused for this implementation at the current time. Some would argue this is an overly early optimization - I would argue it is a good medium term bet based on how many other of these types of templates I’ve worked on in the past, combined with the vast audience I hope can benefit from this automation.
• Crossing a Threshold of Implementation Complexity: Another, but important reason for Pearl Diving is to have a working example best practice template for new solutions. For a long time I have avoided the overhead of creating Custom Resources in CloudFormation - but the solution needs are only getting more complex and so having a pattern for creating all kinds of new Custom Resources is extremely valuable as it will accelerate future efforts substantially.

It is important to know that in past Pearl Diving expeditions I have frequently done a strategic quit on servicing one or more of the Pearl Diving pressures similar to those listed above for this solution. As with Mission Impossible Code heuristics, it seems to be the constraints of the heuristics during experimentation that leads to innovating a solution that fits more of them than expected - but it is hard to predict a head of time what ones will be significantly deoptimized or abandoned.

Pearl Diving Is Better And Faster With an Expert Coach Along

Pearl Diving can be done solo - but it takes much, much longer to work through unfamiliar syntax, object structures and handoffs than if you have a expert coach along! An individuals that help a Pearl Diver must have a coaching perspective because by definition the Pearl Diver will constantly ask questions that are out of their depth. Non-coaching experts will be frustrated with the lack of fundamental knowledge of the technologies in view. Coaching experts - especially with regard to learning code - tend to recognize the desire of someone who is proficient in high level best practices in other languages, will have a natural desire to acquire best practice knowledge quickly - especially the kind that is not easily revealed by search engines. For instance, in this Pearl Dive, it was very challenging to find examples that used boto3 “resources” rather than the more verbose “client” examples. It was hard to even come to an understanding that resources were more appropriate for my implementation needs than clients.

My expert coach in this effort was the Python Wizard Tim Poffenbarger. He patiently offered wisdom and examples to the long sets of questions I had. Huge thanks to Tim!

Super spies make use of specialized tools and techniques when available (and working), but simple and pragmatic alternatives are always top of mind. They jump out of windows, walk across moving cars, use household objects as weapons and drive cars down staircases. They are consistently fashioning situational tools of whatever is found around them. They don’t think of objects and situations as having fixed purposes - but rather that objects and situations are flexible to serve their imposed purposes.

Is it possible to write code that acts like a super spy? Over time I have adhered to a set of coding design heuristics whose parallels to super spy behaviors are an intriguing.

The Super-spy’s hard bent commitment to hyper-pragmatism is counter balanced by an equal commitment to hyper-planning. Every action, every move and every breath will be planned to a tee if and when possible. To the super-spy, planning and opportunism are not seen as two opposite ways to approach a problem, but rather as perfectly complimentary.

A super-spy’s work is never put in an art gallery, nor does it underpin the workings of some great machine. Their work is judged by results alone. While super-spies appreciate beautiful art and love excellent machines - making such things is simply not the skills they bring to bear on their work.

As a coding disposition, pragmatism first, frequently means breaking conventions of coding style, verbosity or slavish adherence to best practices - to go for what works under the most diverse conditions.

Recognize When an Effort is Not a Spy Mission

Inevitably when one speaks with passion and advocacy of a certain approach - more than a few take it as the declaration of a universal manifesto. That the principles discussed are being claimed to apply to all circumstances, for all time. I will tell you right now that I love mission impossible coding. However, that love has also attracted me (self-selected me) to a specific area of experimentation. I am a Cloud DevOps Automation Developer and as such I am constantly called upon to knit together systems and requirements that span many different technologies, contexts and layers.

I admit that super-spies have the same self-selection - they are called upon to knit together a variety of diverse requirements, equipment and people to get the job done. We all intuitively understand that James Bond is the best guy to drive the 2 million dollar super-spy car. But we also understand that Q - not James Bond - is the best one to build the spy car.

Some problems require super-spy field methods and some problems require spy car engineering methods. The methods advocated here might not only fail miserably at building spy cars, most are likely to be anti-patterns to such an effort.

When Possible, Drive on Roads

So if you find yourself repulsed by the methods advocated here because they do not stay within the constructs of a language, do not obey rules for self-documentation or because the solution does not account for proper modularity or other similar details, then, please keep building super-spy cars and weapons, because they are very important in the grand scheme!

Qualities of a Super-Spy

The rest of this series will look at how rugged and opportunistic super-spies engage their work using specific characteristics that ensure they can get the job done no matter what. Some of these are:

• tenacious
• resilient
• resourceful
• opportunistic
• self-sufficient
• self-reliant
• flexible
• pragmatic
• ready
• minimal assumptions
• fault tolerant
• predictive
• perceptive
• precise
• meticulous

Train Your Super-Spy Instincts Using Design Heuristics

Do the above characteristics sometimes conflict with one another? Yes

Is it sometimes difficult to discern which of these should be prioritized in a given situation? Yes

Despite these challenges, do these heuristics generate results? YES

A design heuristic is like a needle on a gauge - it indicates which end of a spectrum of choice to favor - all other things being equal. However, when heuristics are combined, they have competitive tensions in the overall determination of what priorities should be optimized - precisely because all other things are NOT equal when combing them. (A combined set of heuristics are more like connected points in 3D space that define a shape within which are the possible solutions)

To get better at leveraging design heuristics, one must step in and practice - feeling and resolving these tensions again and again. It is not an unnatural learning mode for the human mind and it is supported by the ideas of Agile Discovery. The rewarding result is the training of your “instincts” so that they come up with pragmatic solutions given a complex set of input criteria. Design heuristics also account for the real-world experience that many problems are solvable with multiple different solutions. An approach of rigid laws can imply that there are few solutions or one solution to a given problem.

In the spirit of working examples, a good way to learn about how they are applied in Mission Impossible Code is to take a look at this blog Mission Impossible Code Part 2: Extreme Multilingual IaC (via Standard Code for Preflight TCP Connect Testing a List of Endpoints in Both Bash and PowerShell), paying special attention to the section titled Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

Lesson 1 Right Now - Degrade Your Implementation For Simplicity and Compatibility Reach

Super-spies will quickly and purposely degrade their choices from the “best” or “purpose specific” option to older, at hand options. Don’t have a modern weapon like a gun? Use an ink pen instead!

Here is a case in point - PowerShell’s task scheduling CMDLets versus schtasks.exe. Knitting together three or more PowerShell CMDlets for scheduling a simple task is challenging - it takes hours or days the first time, but it seems to take hours or days to reliably update the same code for newly discovered requirements. In my opinion this is because the CMDLet set reflects the underlying object structures of scheduled tasks a little too directly (A place where PowerShell implementations usually hide the underlying complexity). Consequently scheduling most tasks requires creating 3 or more types of objects with CMDLets and parameters for each. To add to this, these CMDLets are not available on all versions of PowerShell and they are easy to confuse with PowerShell job scheduling CMDlets.

I’m thinking Ethan Hunt would prefer schtasks.exe - it is available on all versions of Windows and most tasks can be scheduled with a single line. For really complex scenarios an exported XML from a working task can be used to setup a new system.

Here is a case in point from: https://gitlab.com/missionimpossiblecode/MissionImpossibleCode/blob/master/ContinueYourAutomationAfterRestartingAHeadlessSystem.ps1

Using PowerShell CMDLets:

$TaskTrigger = (New-ScheduledTaskTrigger -atstartup)
$TaskAction = New-ScheduledTaskAction -Execute Powershell.exe -argument "-ExecutionPolicy Bypass -File $scriptlocation"
$TaskUserID = New-ScheduledTaskPrincipal -UserId System -RunLevel Highest -LogonType ServiceAccount
Register-ScheduledTask -Force -TaskName HeadlessRestartTask -Action $TaskAction -Principal $TaskUserID -Trigger $TaskTrigger

schtasks.exe is more concise to accomplish the same thing:

schtasks.exe /create /f /tn HeadlessRestartTask /ru SYSTEM /sc ONSTART /tr "powershell.exe -file $scriptlocation"

While I am aware there are ways to reduce the number of lines of the CMDLet version, the number of objects does not reduce which means condensing the lines makes the final result much more complex and challenging to build, understand and maintain. It also requires more refactoring when new requirements are discovered.

For me, this is a case where Mission Impossible coding guides me away from the preferred stance of staying within the built-in functionality of one language.

My perspective on scheduling tasks by the most pragmatic methods did not come from a single implementation attempt - I have repeatedly implemented using the CMDlets and repeatedly come up against it’s complexity challenges (both for initial build and code maintenance) and PowerShell version limitations - schtasks.exe simply does not have these problems.

The use of schtasks.exe has these super-spy benefits:

• the resultant code is simple and easy to understand.
• the resultant code has excellent back reach (ScheduledTask CMDLets are available in PowerShell V4 and later).
• when necessary, you can process schedule tasks XML which is readily exported from the task scheduler - so you can use the GUI task scheduler as an code builder for scheduled tasks.

When Super-Spys Make Soup, Ingredients Can’t Be Binary Considerations

Many conversations about technical implementation details start with “Always” or “Never” and quickly cascade into log jams - whether in one mind or in a team discussion. What if you approached creating a new soup recipe with “in” or “out” logic for all the possible ingredients instead of considering how much of each should be used? I am guessing it would be a frustrating experience and the result would likely be disgusting to eat!

Best Practices of technology frameworks are like soup ingredients - the level of each has to be tuned together with the others and with the objectives and scope of the task at hand. When multiple frameworks are involved, the tuning approach is even more important. Like making soup, the right mix is figured out by tasting the various experiments at adjusting the amount of each ingredient.

So heuristics should generally carry intents like “Generally”, “When Possible”, “All Things Being Equal”, “A First Resort”, “A Last Resort”, “First Class Consideration”, etc. instead of “Always” or “Never”.

Heuristics also allow for keeping sight of fundamentals - you can think of this as never losing sight of what a fundamental ingredient like salt does for any soup regardless of what else is in it.

This approach can help reduce architectural tensions and fuel innovative solutions whether being create individuals or teams!

More About Mission Impossible Code Philosophy, Heuristics and Examples

Back to Basics: Testable Reference Pattern Manifesto (With Testable Sample Code) - Mission Impossible Code samples are intended to be both 1) usable directly for production use and 2) a top notch pattern to use for your own innovation.

Continue Your Automation To Run Once After Restarting a Headless Windows System

A Sufficiently Viable Implementation (SVI) for Running Code Under The System Account on Nano Server

Simplicity Manifesto Principles

Principles of Minimalism (with Code)

In my experience, this seemingly small shift in perspective causes a butterfly effect of positive outcomes in the work product. It can be seen in this effort with significant effort around “self-service” and “patching built-in”. In this specific case it also caused ML and AI Ops to be in view because many organizations use scaled CI compute to do model processing runs.

Other things Enablement Automation Code as a Product informed in this case include choosing Boring AWS Technology. In this case, sticking with the “Boring” choice of CloudFormation as the Infrastructure as Code (IaC) language enables thousands of IaC Automation Professionals to quickly implement, innovate and contribute. It may also allow this code to eventually be an AWS QuickStart - a sort of Production-Grade IaC Templates Marketplace provided by AWS.

Studying competing offerings and implementing customer requirements are also hallmarks of product management. At a former employer, I managed a very similar effort specifically for GitLab Runner, where I started to experiment with the Enablement Automation Code as a Product perspective - those experiences have informed this solution.

Enablement Automation Code as a Product strongly relates to a previous blog post on why working examples are also the best learning aide: Back to Basics: Testable Reference Pattern Manifesto (With Testable Sample Code)

Features Checklist

Some features in this list are highlights of especially applicable code inherited from The Ultimate AWS AutoScaling Group ASG Lab Kit, but items marked “NEW” are specifically new compared to that code.

It is important to review the README.md to understand if this code appropriate for your use case.

• Self-Service - “Vending Machine” in the name indicates that there is a purposeful production orientation in developing this code to make it deployable by individuals who are not experts in either AWS or GitLab Runner setup. For instance, there is sufficient commentary in the code and in CloudFormation console forms to enable anyone to figure out how to deploy a runner. Finer points of figuring out a smooth autoscaling metric do require more knowledge in these areas - but standing up an HA runner of any of the types supported is fairly straight forward.
• NEW: Runner Management At Scale - Visibility - runner naming and AWS EC2 tagging are used to ensure it is always easy to know where a runner resides in scaled, multi-account AWS implementations. All runner tags are surfaced as a single comma separated EC2 Tag.
• NEW: Designed for Long Term Runner Management At Scale - Easy Patching and Updates - the “maintainability” built-in to the original Ultimate ASG AutoScaling Group Lab Kit allows runner ASGs to be easily update with all of:
  1. The latest AMI
  2. the OS patches
  3. NEW The latest GitLab runner.
• NEW: Spot / Ondemand Compute Type Surfaced as Runner Tags - CI/CD Automation developers can choose what type of compute is acceptable for their purposes at the per-job level of granularity.
• NEW: CloudWatch Instance Metrics Collected for Linux and Windows - mainly for Memory based scaling, but disk and network are also collected so that bottlenecks with these resources can be spotted for specialty workloads like ML Ops. They are also dimensioned on “Instance Type” for the same reason - analyzing bottlenecks by instance type in order to select the best one depending on workload.
• NEW: Scaling on Memory Utilization - workloads that have low CPU utilization, but still consume memory (for example GitLab jobs that just poll for status from another system), may be better scaled using memory utilization.
• NEW: Runner On/Off Scheduling - one stop and one start schedule (provided by ASG scheduled actions) are provided to completely shut the runner down when not being used. Use cases include: a development team specific CI runner that is used only when developers are on working hours, CD runners that are only needed during deployment events to specific environments, runners that are manually shutdown when done - but need to auto-start at a given time of day or week.
• NEW: Provides Linux “docker” Runner Executor for Primary Use Case of docker+machine Scaling Replacement - docker+machine GitLab Runner executor is deprecated because docker machine is deprecated by docker. This replaces the docker level scaling with
• NEW: Provides Windows shell Runner Executor for Primary Use Case of .NET Framework and Other Windows Development - some .NET Framework CI builds and other Windows CI builds may require a full Windows instance due to the build tooling requirements. By having a Windows shell runner that can also auto-scale, development teams with these requirements can have a scaling GitLab runner.
• NEW: Provides Linux Shell and Windows Docker GitLab Runner Executors - while less common, these are provided as well.
• NEW: Extensive Troubleshooting Information Documented - [TESTING-TROUBLESHOOTING] (https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/master/TESTING-TROUBLESHOOTING.md) - also linked from README.md
• Designed for Extensibility and Testability - the runner configuration scripts are (a) separate from the cloud formation and (b) downloaded dynamically at scaling time. This enables getting around some code limitations of CloudFormation, but it also enables others to easily create their own runner configurations and store them anywhere. It also enables iterative testing over just the runner script portion by scaling down to 0 and then back up - the dynamic sourcing of the scripts causes updates to be taken. For stronger version pegging or “full immutable automation code” the script can have a version number embedded in the file name and be source from S3.

Code for This Post

GitLab HA Scaling Runner Vending Machine for AWS

Mission Impossible Code Series Inclusion

• The solution sticks to the Boring Technology selection criteria.
• The solution is implemented in a single CloudFormation template.
• The solution implements “Least Privileges”
• The solution implements “Least Configuration” (don’t configure things that the user indicates they won’t use, use blank parameters as an off switch for least config).

Maybe you have a team of Windows developers that are onboarding for your new Git server installation or maybe you’ve decided to drop http password authentication to your existing Git server (due to it’s many problems). Your next steps may well be into a rough and rocky rabbit hole when you were holding out hope for simplicity (you know the kind you’ve fallen into before if you’ve been in tech for more than about 45 minutes).

The guides on the internet for getting Windows setup for SSH authentication for Git are unnecessarily complex.

My inner tool smith really loathes when the very first steps into something new are fraught with rocky rabbit holes - so I took on the challenge of creating an easier way.

The resultant tool is a 20 line PowerShell script that deploys Git, configures SSH and leaves the public key on your clipboard so you can paste it into GitLab or any other Git collaborative webserver. There is also an optional connectivity test.

Reasons For Moving to SSH

There are multiple reasons you may want to move your Windows developers to SSH authentication for Git:

1. You want to get away from git storing local passwords - whether in the git config or in Windows Credentials (with the windows credential helper) because it is pure pain to walk people through how to find and update this password when they change it on the Git server.

2. You want to avoid both http passwords and the http protocol for git.

Conventional Wisdom on SSH Configuration

The conventional wisdom solution offers many steps that are roughly:

1. Installing git manually.

2. Installing the well known Windows SSH client Putty.

3. Installing Putty’s key generator.

4. Converting the non-compatible putty generated key into an ssh compatible one.

5. Precisely placing the SSH key on disk.

6. Precisely permissioning the SSH key and it’s parent folder (ssh is purposely fussy about this in order to keep the key secure).

Most of this can be avoided by simply using the full SSH client that is embedded inside of the Windows git client install.

The Cleanest Way (With Working Automation Code)

Besides the above pure pain, here are the additional things solved for in this code:

1. Automatically installs Git - but only if necessary (idempotent)

2. Automatically installs chocolatey to install Git - but only if necessary (idempotent)

3. Automatically generates an SSH key - but only if necessary (idempotent) (which avoids killing a key that might be in use)

4. Uses the Git’s built-in SSH client to create SSH keys (avoids the complexity of the above conventional wisdom)

5. Copies the public key to the clip board and pauses for the user to add it to the Git server (in their profile)

6. Optionally does a SSH login test if you provide a value for: $SSHEndPointToGitForTesting

Solution Details

This code can be run directly from GitLab with this command:

Invoke-Expression -command "Invoke-WebRequest -uri 'https://gitlab.com/missionimpossiblecode/MissionImpossibleCode/-/raw/master/install-gitwithssh.ps1' -UseBasicParsing -OutFile ./install-gitwithssh.ps1" ; . ./install-gitwithssh.ps1

If you want to download dynamically, but also want the test and instructions to work, then set these environment variables before calling the above:

$env:YourGitServerhttpURL="https://gitlab.com" $env:GitSSHUserAndEndPointForTesting="git@gitlab.com" 
#some Git servers might want the windows userid "git", which is specified as $env:username

You can also simply copy the code, hardcode the two variables and distribute it in your organization.

Main Code

# Set environment variables before calling in order to test 
If ((Test-Path env:YourGitServerhttpURL) -and (!(Test-Path variable:YourGitServerhttpURL))) {$YourGitServerhttpURL="$env:YourGitServerhttpURL"} If ((Test-Path env:GitSSHUserAndEndPointForTesting) -and (!(Test-Path variable:GitSSHUserAndEndPointForTesting))) {$GitSSHUserAndEndPointForTesting="$env:GitSSHUserAndEndPointForTesting"} 

# $YourGitServerhttpURL="https://gitlab.com" 
# $GitSSHUserAndEndPointForTesting="$env:username@gitlab.com" #Optional to trigger testing Use "git@gitlab.com" for GitLab.

If (!(Test-Path 'C:\Program Files\git\usr\bin\ssh-keygen.exe')) { Write-Host 'Installing latest git client using Chocolatey' If (!(Test-Path env:chocolateyinstall)) { Write-Host "Chocolatey is not present, installing on demand." iwr https://chocolatey.org/install.ps1 -UseBasicParsing | iex } cinst -y git } 
If (!(Test-Path $env:userprofile.ssh\id_rsa.pub)) { Write-Host 'No default ssh key present in $env:userprofile.ssh, generating a new one.' Write-Warning 'Press enter for default file name and twice for password to set it to not have a password' & 'C:\Program Files\git\usr\bin\ssh-keygen.exe' } get-content $env:userprofile.ssh\id_rsa.pub | clip write-host "Your public ssh key is now on your clipboard, ready to be pasted into your git server at $YourGitServerhttpURL"

If (Test-Path variable:GitSSHUserAndEndPointForTesting) { Write-Host 'NOTE: Sometimes it takes a while for your Git server to propagate your key so it is available for authentication after first adding it!' Write-Host 'After you have setup the key, to test the connection, press any key to continue...'; $null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown'); 

#Use git's open ssh: 
Write-Host "...Testing ssh login as ${GitSSHUserAndEndPointForTesting} using key $env:userprofile.ssh\id_rsa on port 22" $env:term = 'xterm256colors' push-location 'c:\program files\git\usr\bin' .\ssh.exe "${GitSSHUserAndEndPointForTesting}" -i $env:userprofile.ssh\id_rsa -p 22 pop-location Write-Host 'After observing the test result above (note it may take time for your new key to propagate at the server), press any key to continue...'; $null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown'); }

Code For This Article

install-gitwithssh.ps1

Mission Impossible Code Series Inclusion

• The solution is very concise.

• The solution is idempotent (only takes steps necessary when things are missing).

• The solution solves multiple challenging issues in the simplest possible way.

So Jefferson and I worked through this approach. Before the session I played with some ideas and approaches, but ultimately we ended up with a new approach.

Lessons Learned:

• GitLab has linting as a part of it’s built-in Code Quality scanning:
  • But it can be a challenge to figure out that this is where GitLab CI does linting - you have to dig into the website of the underlying software “Code Climate” to figure out that this is being done and for what languages.
  • GitLab’s built-in support is configurable, including adding your own Code Quality scanning jobs with additional products.
• Super-Linter has a lot of language coverage.
• Super-Linter is implemented in a container - this means implementation on GitLab CI should be even easier - especially if the image built for it is publicly available. If the image was not publicly available, we would have built the docker file on GitLab as a prerequiste to using it in GitLab CI.
• Super-Linter is driven by a lot of variables - very compatible with GitLab CI.
• Super-Linter has a special parameter, RUN_LOCAL, that overrides a bunch of expected GitHub Actions input parameters - so we don’t need to emulate or map those expected parameters. Perhaps many GitHub Actions can be repurposed easily using this parameter.
• Super-Linter only outputs results to the log - not as an artifact file. Hopefully as it matures it will be able to emit test results. If it could emit JUNIT.xml we could have loaded the results into GitLab’s test results visualization panels. If we could transform the results into GitLab Code Quality result format, results could be even more deeply integrated inline into Merge Requests.

Techniques Demonstrated:

• Examing the source code and documentation for clues on how to implement Super-Linter outside of GitHub.]
• Iteration - Let GitLab CI “give it a shot” while we research other things.
• Moving variable declarations around to ensure they are encountered early enough (a final break through by Jefferson was to move a key variable declaration from the script into GitLab’s variables: block)
• Make Into a Plug-in Extension by:
  • Reimplementing the live code in a way that enables it to be simply included in other CI.
  • Provide an example of calling the plug-in.
  • Demonstrate how to version peg the plug-in using GitLab’s include:ref: to point to a specific git tag of the plugin.

The Code

• GitLab CI CD Plugin Extension GitHub Action Super-Linter - as completed in live coding
• GitLab CI CD Plugin Extension GitHub Action Super-Linter - as productionized
• Example of Calling the Super-Linter GitLab Extension

References

• GitLab Built-in Code Quality Scanning
• Which uses CodeClimate, here are the Supported Scanning Engines, Including Linting
• Implementing a Custom Code Quality Tool in GitLab

Mission Impossible Live Coding

The past recordings, including the above are at: https://www.youtube.com/channel/UCcL7kaon80MiwuGcNI7A1Dg

We are live coding every two weeks on Friday at 2pm ET - the next one will be on July 3rd.

You can add it to your calendar with this .ICS file.

The live stream will is at: https://www.twitch.tv/missionimpossiblecode and https://www.youtube.com/channel/UCcL7kaon80MiwuGcNI7A1Dg.

We will be using GitLab issues to queue up questions and enable you to submit code snippets for consideration. Please use this GitLab project for that purpose: https://gitlab.com/missionimpossiblecode/live/-/issues

Why? Two main reasons: In a world of stripped back distros (think containers), even fundamental Gnu coreutils and other basics can be missing. Which leads to the second frustration - for some reason distro families thought it would be great to not only innovate package management technology but also change up the command set and also not provide a universal command set (like an api) to hide the differences.

So after bashing my head on this a million times, a bit of bash code eventually emerged (yeah from my head). And as you’ll read, it was not a process of random chance and natural selection - but rather it’s opposite - hyper-engineering.

As per Mission Impossible Code Principles I’ve tried to make it “as simple as possible, but still have a very broad scope of reuse”

FYI - Windows isn’t any better - while Windows PackageManagement (aka OneGet) and Chocolatey both tried to consolidate down to one command set for all package types for the platform, we now have a third one.

TL;DR

I used to take time to expand the section “Architecture Heuristics” and discuss the enablement created by each of the bullets in the section.

I won’t be doing that any longer for these reasons:

1. I have added “Benefits” and “Coding Decisions” to each of the below to provide a synopsis of how the item benefits the solution and what coding choices it affected - which provides a better information map for faster learning.
2. By providing this learning section as mapped bullets - you can incrementally learn a few points by scanning and deep diving what interests you without feeling obligated to plow through paragraphs of prose.
3. It’s quicker and easier for me to document without having to generate formal copy - I’ll be more tempted to share when the authoring process is lighter.

Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

This section appears in many Mission Impossible Code samples because it: 1) Demonstrates there is an actual pattern to the thinking underlying this approach, 2) It allows you to learn the patterns but making the thinking plain, 3) It shows the complex engineering of creating simplicity.

The following list demonstrates the Architectural thrust of the solution. This approach is intended to be pure to simplicity of operation and maintenance, rather than purity of a language or framework or development methodology. It is also intended to have the least possible dependencies. It’s an approach I call “Mission Impossible Coding” because it enables the code to get it’s job done no matter what.

• Requirement: (Satisfied) Adhere to “Adoption Driven Development for Tooling”
  • Benefits: everyone will use more of your stuff, more people will truly ’love’ your stuff, driving implementation decision toward “human cognitive” priorities over machine considerations - because we all have much less time than computing resources
  • Coding Decisions: Adhere to Mission Impossible Techniques (aka the entire Architecture Heuristics section)
• Requirement: (Satisfied) Don’t rely on errors to do detections (passive detection)
  • Benefits: creating automation errors when formal exception checking needs to be supported
  • Coding Decisions: picking “command -v” to detect existence of a command
• Requirement: (Satisfied) Don’t require prerequisites to work
  • Benefits: avoid chicken and the egg problem, works on minimalized containers images
  • Coding Decisions: picking “command -v” to detect existence of a command
• Requirement: (Satisfied) Be compact.
  • Benefits: Enables “single script” implementations as the code is not overly obnoxious at the top of a script. Many orchestration systems can send a single script to an endpoint, but sending files makes their scenario much more complex - compact code can travel inside the single script.
  • Coding Decisions: IFS + read method of parsing key value pairs, some single line if statements
• Requirement: (Satisfied) Ensure parameters can be passed in from an unknown number of parent levels and unknown technologies or computing languages
  • Benefits: very broad reuse with no adaptations for many levels of enclosing orchestration.
  • Coding Decisions: Work with a 1) list that is 2) constructed as a string (so that it can be passed between all types of automation technology).
• Requirement: (Satisfied) Handle at least yum and apt-get.
• Requirement: (Satisfied) Be idempotent by checking for the existence of a command before attempting installation operations.
  • Benefits: all the benefits of idempotency which are too many to list here.
• Requirement: (Satisfied) Work for scenarios where the name of the package is different than the name of the command that is needed.
  • Benefits: broader reuse due to handling this frequent use case
  • Coding Decisions: use key value pairs for installing packages to get specific commands
• Requirement: (Satisfied) Auto-detect the package manager.
  • Benefits: broader reuse
• Requirement: (Satisfied) Handle the AWFUL problem of brew taps and casks (who built that? - can someone spend a day and abstract away this useless distinction right in the brew code like I’ve done here :) - yeah and disallow a package identifier to be both while you’re at it ;) )
  • Benefits: broader reuse, declarative state approach (just “make it so”)

The Code

New versions of the code are not synced to the below article insert, please use the repository link below to get the latest.

function ifcmdmissing-instpkg () {
#If you don't need brew, this can be much more compact by removing the relevant code
#If command is not on path, installs the package
#detects package managers: apt, yum and brew
  if [[ -n "$(command -v brew)" ]] ; then
    #Detect package manager, brew first because some macOSes have an apt-get stub
    PKGMGR='brew'
  elif [[ -n "$(command -v yum)" ]] ; then
    PKGMGR='yum'
  elif [[ -n "$(command -v apt-get)" ]] ; then
    PKGMGR='apt-get'
  fi
  for cmdpkg in $1 ; do
      IFS=':' read -ra CMDPKGPAIR <<<"${cmdpkg}"
      CMDNAME=${CMDPKGPAIR[0]}
      PKGNAME=${CMDPKGPAIR[1]}
      echo "If command ${CMDNAME} is not on path, the package ${PKGNAME} will be installed."
      if [[ -n "$(command -v ${CMDNAME})" ]]; then
          echo "  '${CMDNAME}' command already present"
      else
          echo "  Missing command '${CMDNAME}'"
          echo "  Installing package '${PKGNAME}' to resolve missing command."
          if [[ $PKGMGR != 'brew' ]]; then
            if [[ $PKGMGR == 'apt-get' ]]; then $PKGMGR update; fi;
            $PKGMGR install -y ${PKGNAME}
          else
            #automatically abstract the difference between brew taps and casks
            if brew info ${PKGNAME} >/dev/null 2>&1; then
              brew install ${PKGNAME} --force
            elif brew cask info ${PKGNAME} >/dev/null 2>&1; then
              brew cask install ${PKGNAME} --force
            else
              echo "  '${PKGNAME}' not found as a formulae or cask"
            fi
          fi
      fi
  done
}

ifcmdmissing-instpkg "jq:jq wget:wget curl:curl"

Tested On

• MacOS
• Ubuntu container
• Amazon Linux 2 container

Source Code For This Article

ifcmdmissing-instpkg.sh

Appendix: Mission Impossible Pattern Philosophy

Mission Impossible Code samples are intended to be both 1) usable directly for production use and 2) a top notch pattern to use for your own innovation. More details on this approach are in the post

Appendix: Constructed Simplicity (Conciseness) is The Tip of An Iceberg

While epitomized by the Blaise Pascal quote “I have made this longer than usual because I have not had the time to make it shorter.”, I find that the process of creating concise, simple observations and solutions is complicated and sometimes complex.

Creating concise designs and solutions is a deep passion of mine. However, there is a frustration that is quick on the heals of creating something that is concise. In my line of work, at some point, you usually have to justify your final work. That is the point at which the rest of the iceberg of thinking that backs the concise tip comes to the fore. Frequently the reaction is that it can’t possibly be that involved or that you are spinning up reasons on the fly to simply bolster an idea or solution that was arrived at haphazardly.

Why bother addressing this sentiment? Because concise, mission impossible style, solutions can easily be criticized as lacking sophistication in their implementation. But much like a Mark Twain quote - that ruddy external appearance belies a hard wrought balance of many interrelated tradeoffs to get to a simple solution.

Said another way, solutions that are earnestly designed for conciseness can be rewarded by a perception of being the opposite of what they are - simplistic, backed with little or no thought.

This Mission Impossible Coding series exposes the submerged methodological icebergs below the waterline of the visible and concise solutions it attempts to arrive at.

The first is to have instances self-tag themselves as to whether they are a spot or on-demand instance. When supporting a mixed instances policy, the implementation requires a little more thought.

A second is the ever common need for a bucket that the ASG instances have access to - whether for AWS SSM results collection or inventory, for deployment artifacts to update instances, access to data or many other uses.

While a relatively small change, the title has also been updated to “The Ultimate AWS ASG Kickstart and Lab Kit” to indicate it is appropriate for experimentation and also as the foundation for a deployable configuration.

I was able unable to avoid the temptation to sneak in a couple other improvements as well.

TL;DR Feature Summary

Skip learning new ideas and go straight to the CHANGELOG.md If you missed the original article it is here: The Ultimate AWS AutoScaling Group ASG Lab Kit

New Features Checklist

• Instances tag themselves as spot or on-demand.
• S3 Bucket Setup
• Managed Permissions Instead of In-line
• Optionally provide a keypair name for remote access.
• Use Cloudformation “Rules:” for cross parameter validation.

Features in Detail (Ideas)

Instances Tag Themselves as Spot or On-Demand:

One of the downstream uses I intend for this Kit is to build a GitLab CI Runner on it and I realized that ephemeral compute is good for some CI workloads which are more interruptible - like mass automated testing - but not good for others where the impact of interruption might be much more significant - like long running deployment processes. By allowing instances to self-identify and be selectable by CI engineers, they can make their own decisions about what CI workloads to run on ephemeral compute.

The resultant Instance tag is either COMPUTETYPE=SPOT or COMPUTETYPE=ONDEMAND. The variable $COMPUTETYPE is available throughout the userdata script including if you are using the template to pull in your own userdata code. By leveraging this variable you can surface this data to other systems you may be installing on the instance. For instance, I will be adding a GitLab CI Runner tag with this data.

An interesting detail here was that attempting to tag by propagating from the ASG LaunchTemplate won’t work for a mixed instances policy - when the LaunchTemplate might launch a mix of spot and on-demand, it is important that the instance self-detect and self-tag.

Many internet searches directed me to use the aws call “describe-spot-instance-requests”, however, in the spirit of Least Privilege and Least Config - I dug a little deeper to find that it could be done purely with “describe-instances”. Since this template already required describe instance permissions to handle ASG Lifecycle hooks, using “describe-instances” meant that I could use familiar code and not have to add permissions

S3 Bucket Setup

As with many other resources this template can either an auto-create a bucket for you, or you can override bucket name with one that exists. In either case, Permissions are applied to the instance profile.

Managed Permissions Instead of Inline

Previously the template created permission using inline IAM Policies attached to the created instance role. This meant they could not easily be attached to existing roles. By creating IAM Managed Policies, they are easily attached to any existing role. They are also created even if you do not have the template auto-create the IAM Instance Profile Role.

Use CloudFormation “Rules” for cross parameter validation

It is a common mistake (at least for me) to choose “Windows” as the platform, but then forget to update the SSM parameter path for image lookup to point to a Windows image. Unfortunately you don’t discover this until after the machine starts. Now the template prevents execution when your parameters are in this state. It also serves as a working example of CloudFormation rules, which hard to come by - especially in YAML.

Code for This Post

CloudFormationUltimateAWSASGLabKit.yaml

Create Now in CloudFormation Console

Due to the frequency of needing to do it, one of the biggest challenges is tailing an eventlog while waiting for results.

When following a text log, I simply use Get-Content logfilename -wait to emulate the Linux command tail -f logfilename

So I went in search of what I thought would be a quick find, but all my finds were all way to long and involved - so I made a new oneliner that follows the principles of Mission Impossible Coding.

Managing Windows using only the PowerShell Console affects a growing list of Windows deployment scenarios:

• Server Core - no GUI available.
• Windows Containers - no GUI available.
• Using a Cloud Shell like AWS SSM Session Manager, Azure Cloud Shell or Google Cloud Shell.
• PowerShell Remoting.
• VS Code Remote Development.

In all of these situations, tailing various Windows Eventlogs is an essential capability for development debugging and operations troubleshooting.

Most of the existing approaches use message indexes - which have to be retrieved by calling an API - and then tracks the last retrieved index. This means no oneliners - lots of multiline functions and full blown CMDLets.

I realized that for my purposes really old log lines were not of interest - even if they were the last 5 to be received - it was more about the most recent ones within a timeframe I was interested in.

So I noodled whether I could use time, rather than the message index of the specific log.

It turns out that you can use time - and rather than looking back a specific number of index entries when first loading, I look back a certain number of minutes.

This vastly condense the code to the point of being reasonable oneliner.

Why do I care about a oneliner? I had been testing the termination lifecycle hook The Ultimate AWS ASG Lab Kit - and each time I perform a test it is on an ephemeral instance that just booted and I actually issue a termination command to see the code working. So any amount of fussing installing modules or using vast tracts of code for simple functions is painful.

I’m going to assume that many of you working on CLI only Windows (whether through a remote or cloud shell or whether using containers) can also appreciate the power of a oneliner in these situations.

The code is below, but a quick walk through is:

1. Define it as a function since it is few extra characters and I can use it again (though I have provided a listing without the function below).
2. By default, look 5 minutes into the past (set $lastdate).
3. Setup a loop that goes until it gets a CTRL-C
4. Set $newtime (can’t dynamically use Get-Time or we risk losing events during the loop)
5. List the events between the times.
6. Set $lasttime=$newtime
7. Loop again.

Enjoy!

The below command emulate this command on Linux:

tail -f /var/log/messages

PowerShell Oneliner with Function (Can set which log and how many minutes to look back in initial output and computername):

Function Tail ($logspec="Application",$pastmins=5,$computer=$env:computername) {$lastdate=$(Get-date).addminutes(-$pastmins);while ($True) {$newdate=get-date;get-winevent $logspec -ComputerName $computer -ea 0 | ? {$_.TimeCreated -ge $lastdate -AND $_.TimeCreated -le $newdate} | Sort-Object TimeCreated;$lastdate=$newdate;start-sleep -milliseconds 330}}; Tail

Smaller PowerShell Oneliner hard coded for 1) the Application log, 2) 5 minute lookback and 3) local computer only. (at 243 characters, it is 116 characters (30%) shorter than the 360 character oneliner above):

$lastdate=$(Get-date).addminutes(-5);while ($True) {$newdate=get-date;get-winevent Application -ea 0 | ? {$_.TimeCreated -ge $lastdate -AND $_.TimeCreated -le $newdate}| Sort-Object TimeCreated;$lastdate=$newdate;start-sleep -milliseconds 330}

P.S. This post has been included in the Mission Impossible Code series because:

• The solution is very concise.
• The use of dates versus eventlog index seems to be a new approach (that enabled much shorter code than using index).
• It is pragmatic and efficient to the need at hand.
• As a oneliner it is easy to bring to ephemeral test machines.
• It has enough features to be used as a complete solution for log tailing.
• It supports remote computers.

I had a number of improvements I wanted to make to this template set and this blog represents that work.

The result is really the answer to the question “What would be a minimal, but production-useful working example to learn and experiment with AWS ASGs that use spot instances and proper lifecycle hooks?”

Since the last team I managed had to do all of our automation work for both Windows and Linux, I wanted the solution to work for both.

TL;DR Feature Summary

So here is the net set of functionality that the Ulitimate AWS AutoScaling Group Lab Kit includes:

Previously Existing Features

The following features were made available in a previous incarnation of this template at: https://MissionImpossibleCode.io/post/asg-lifecycle-hook-for-linux-kernel-patching-with-a-reboot-in-aws-autoscaling-groups/

• Create a “Launching” ASG lifecycle hook so Linux kernel patching could reboot before health checks start (thereby avoiding early termination).
• Allow an test web app to be installed to emulate a real application server.
• Maintainability Built-in: Enable patch updating of the cluster by simply updating the CloudFormation stack.
• Optional Troubleshooting mode that sets up SSM permissions and installs the SSM agent.
• Support High Availability Only (no scaling): Warm HA (1 Instance - usable for applications that don’t support multiple nodes), HOT/HOT HA (2 instance ASG - if application supports it)
• “logit” function to expose script information to the console and common logs (Linux: /var/log/messages, Windows: Application log

Here is the previous article if you want to learn more about it’s features and design - which includes comparison to other ASG patching methods: ASG Lifecycle Hook for Linux Kernel Patching with a Reboot In AWS Autoscaling Groups

New Features (all for both Windows and Linux):

• Be a great lab kit for learning, but also be a great starting point for actual production implementations.
• Windows support for all previous functionality which was only for Linux. This is especially important if your Windows spin up and initial automation might exceed the default hook time of 60 minutes (ahhh, and time to make a custom AMI for that and to never use T2 instances - just saying)
• Maintainability Built-in: Allow CloudFormation to lookup the latest AMI (including on updates) AND enable an override to peg to a specific or custom AMI.
• Optional installation of CodeDeploy in case the ASG is wired to it for code deployment.
• Group CloudFormation parameters in a sensible way, rather than the default alphanumeric sort.
• Support “Least Resource Creation” by only creating AWS resources when they will be used - for instance not configuring SSM IAM permissions if the troubleshooting feature was not configured.
• Support Spot Instances and basic spot configuration parameters.
• Support non-Spot configurations (by setting On Demand Percentage Above Base to zero) - which also supports complete on-demand ASGs that can select from multiple instance types to avoid failure when a specific instance is exhausted in an availability zone.
• Support Configurable Autoscaling (optional) and include parameters for configuring it (step scaling policies)
• Support TERMINATING lifecycle hooks and cleanup script for implementations that should do clean up or deregistration when the ASG scales in.
• Support three OS Patch Scopes: all patches, only security patches and no patching (for faster testing of other things)
• State based installs - only trigger installs if the desired software is not present already.
• Built-in scaling testing by including an optional sythentic CPU driving utility and SSM parameter to control it. This allows you to dial-in the CPU utilization load you want the ASG to be under and change it after deployment to completely validate the scaling parameters and smoothness. Following the “Least Resource Creation” principle, the resources to support this are only deployed if you configure the capability.
• Allow override of the basic, built-in Instance Profile IAM Role that the template creates with one that already exists.
• **Allow extension of userdata from an embedded script, local file or a file to download from s3://, http:// or https://

Technical Design

Minimal but Completely Working Template

The CloudFormation template is purposely minimal in order to more clearly demonstrate the concepts of the solution. At the same time it includes everything needed and works. The approach adheres to The Testable Reference Pattern Manifesto

Tested With Both ASG Updatepolicy Settings

The parameter UpdateType defaults to “RollingThroughInstances” which sets the UpdatePolicy to use AutoScalingRollingUpdate, but it can be changed to “ReplaceEntireASG” to set the UpdatePolicy to use AutoScalingReplacingUpdate. Although not tested with Lambda based updates, they would be expected to work just fine with this template.

Least Privilege IAM

The IAM Roles and least privilege permissions are included so that it is clear what permissions are needed and so that instances do not have more permissions than needed to interact with their own ASG. Two possible methods for limiting the permissions are provided. Using the ASG name in the Resource specification of the IAM is active. Using a condition on a tag is provided as a tested, but commented out alternative.

Maximizing ARN Flexibility for Template Reuse

The ASG arn in the IAM policy with the SID “ASGSelfAccessPolicy” demostrates maximizing the use of intrinsic AWS variables by using them for AWS Partition (use in Gov cloud or China without modification), AWS Account ID (use in any account) and AWS Region (use in any region without modification).

Works Without ASG

If the userdata code cannot retrieve it’s ASG tag it assumes that it is not in an ASG and all lifecycle hook actions are skipped. This allows the solution to be used in non-ASG scenarios.

Patch Maintenance Built-in

Zero-downtime patching for the entire ASG is supported by updating the PatchRunDate in the cloudformation stack - the entire fleet will be replaced with instances that are up to date on patching. The date is purposedly used to record an environment variable within Userdata so that the ASG Updatepolicy knows it should replace all instances.

Scheduled ASG Patching

By simply scheduling a cloud formation update command with an updated date, the entire ASG will roll. The most AWS cloudy way to do this is a scheduled CloudWatch Event that triggers a Lambda function.

aws cloudformation update-stack --stack-name "your-asg-stack" --parameters ParameterKey=1OSPatchRunDate,ParameterValue=$(date '+%Y-%m-%d'),UsePreviousValue=false

Scheduling Instance Availability

If you are using this template primarily for HA for an instance, you can also consider using skeddly to set the ASG Desired and Minimum counts to zero for the hours that the instance will not be in use. This assumes that the installed software has it’s state data somewhere else and that you use the termination monitoring to perform any orderly application shutdown if it is needed.

Dynamic Extension of Userdata

Added in Version. 1.2.0 Allows additional script commands during startup. This is parameterized for testing new versions and to enable one CloudFormation template codebase to be used for many different Autoscaling groups. It also allows you to use this template without customizing it so that you can take future updates without headache. Windows 2012 and earlier also have a userdata size limit of 16Kb - this method gets around that.

1. “Embedded” uses the code right in this template and does not use external files at all.
2. Enter a URL starting with s3://. s3 allows easy private file storage.
3. http:// or https:// to dynamically source one during instance provisioning. http/s enables usage of git raw urls (whether public or private).
4. Enter a file pathname on the local instance. The file must be present in the location by the time Userdata processes (e.g. via a custom AMI)

For all external file sources, the instance must have a network route and permission to any remote locations.

The code you write must be idempotent so that it does the correct thing when run again after a patching reboot.

There is a simple example at: https://gitlab.com/DarwinJS/ultimate-aws-asg-lab-kit/-/raw/master/CustomInstanceConfigurationScriptSample.sh_and_ps1

Monitoring and Metrics

Two monitoring and metrics values are recorded as metadata. You can control what log file the is added to (or mute the log file) by altering the function “logit”. Generally you want this to be a log file that is collected by your log aggregation service (sumologic, loggly, etc). If you already collect /var/log/cloud-init-output.log, you can mute the log file write to /var/log/messages.

LAST_CF_PATCH_RUN

The CloudFormation parameter PatchRunDate is:

• saved on the instance as the environment variable LAST_CF_PATCH_RUN in /etc/profile.d/lastpatchingdata.sh
• emited to /var/log/messages as “LAST_CF_PATCH_RUN: ”
• added as a tag to both the ASG and all Ec2 instances

This date simply indicates the initial setup of the ASG or the last fleetwide forced patch. It also serves to purposely change something in userdata so that the entire fleet is forced to be replaced when you run an update and change this date.

ACTUAL_PATCH_DATE

The date as of spin-up is:

• saved on the instance as the environment variable ACTUAL_PATCH_DATE in /etc/profile.d/lastpatchingdata.sh emited to /var/log/messages as “ACTUAL_PATCH_DATE: ”

Instances that spin up as a result of autoscaling will not have their patches limited to the date expressed in LAST_CF_PATCH_RUN, so ACTUAL_PATCH_DATE tracks the date they were actually patched.

Comparing these two dates can help you understand if you have developed a large variety of patching dates due to autoscaling and might want to roll the fleet to a standard date by updating the cloudformation with a new PatchRunDate.

Testing and Observing

Kicking Off The Template

Use the AWS CloudFormation console to launch the template - to see how subsequent updates will work, pick 4 instances and set TroubleShootingMode to true.

Testing Scaling Configuration with Synthetic CPU Loading

You can validate whether the following respond as designed:

• Verify designed scaling responsiveness and smoothness - up and down.
• Verify AZ scaling configuration.
• Verify Spot / On-demand instance parameters are responded as designed including instance types, mixed instances policy, percentage spot, etc.

During deployment, be sure to enter a numeric value for the 8DBGCPULoadPercentInitialValue parameter (Yeah sorry, I even like my variable names to be fully self documenting).

If you do not want scaling to occur immediately, set it low to something like 5.

If you do not provide a value at all, Synthetic CPU Loading is not even setup because this template follows a principle of “Least Configuration”.

After the template completes, you will find a new SSM parameter that is named as “YourASGName-SyntheticCPULoad” as the parameter name. Since the ASG name is dynamically named it will be prepended with some random characters.

You can now vary the synthetic CPU load using the parameter and watch the CloudWatch alarms for scale out and scale in and watch the AutoScaling Group for scaling actions.

IMPORTANT! Do not deploy the template with a value that causes scale out and then forget about it for a long period or overnight - you might bankrupt your company with AWS billing charges.

Observing Lifecycle Hooks in AWS Console

In the EC2 Console open the Autoscaling group, on the “Lifecycle Hook” tab observe the ‘instance-patching-reboot’ hook is configured.

Also, before the instances are in service you can see “Not yet in service” in the “Activity History” tab and “Pending:wait” in the “Lifecycle” column of the “Instances” tab for each instance. These will change to indicate the instances are in service as each instance completes setup procedures.

The same is true for observing the terminating hook.

Observing On Instance Script Actions

All the actions of this template can be observed without logging into the instance by using the AWS console to view the system log for instances (Right Click Instance => Instance Settings => Get System Log) and scanning for the text “USERDATA_SCRIPT:”

The first message will contain “Processing userdata script on instance:”. All the messsages include timestamps so that you can observe things like how long a reboot took and the fact that if you don’t sleep the script, it keeps processing for a while after the reboot command.

On Windows you would need to retreive the Application log to watch launching and terminating hook actions.

If you enable the debugging mode you can get a web based console prompt on both operating systems using SSM Session Manager console.

Observing Logs on The Instance

If you need or want to logon to the instance for examination or troubleshooting, set the parameter TroubleShootingMode to ’true’. This enables SSM IAM permissions and installs the SSM agent on the instances to allow AWS Session Manager to logon using SSH or WinRM. For linux, the log lines that you see in the AWS System Console will be in the CloudFormation log at: \var\log\cloud-init-output.log. For Windows it will be the Application log. For observing the termination hook SSM will leave the last received log on the screen - so you can actually see the termination messages after the instance is gone. On Linux you can use:

tail -f /var/log/messages

On Windows you can use (it is also a good generic EventLog tailing function):

Function Tail ($logspec="Application",$pastmins=5,$computer=$env:computername) {$lastdate=$(Get-date).addminutes(-$pastmins);while ($True) {$newdate=get-date;get-winevent $logspec -ComputerName $computer -ea 0 | ? {$_.TimeCreated -ge $lastdate -AND $_.TimeCreated -le $newdate};$lastdate=$newdate;start-sleep -milliseconds 330}}; Tail

Observing Pseudo Web App

If you set SetupPseudoWebApp to true, the following is done: 1) A port 80 ingress is added to the default VPC security group, 2) Apache is installed, 3) an apache home page is created which publishes the patching and ASG details of the

Code for This Post

CloudFormationUltimateAWSASGLabKit.yaml

Create Now in CloudFormation Console

Nitty Gritty Appendix: Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

This section details some of the complex journey that goes into creating a simple and highly functional pattern. It is most helpful to those who would like to architect on top of this solution.

I find it very helpful to enumerate architecture heuristics of a pattern as it helps with:

1. keeping track of the architecture that emerged from the 'design by building' effort.
2. my own recollection of the value of a pattern when examining past things I've done for a new solution.
3. others quickly understanding the all the points of value of an offered solution - helping guide whether they want to invest in learning how it works.
4. facilitating customization or refactoring of the code by distinguishing purpose designed elements versus incidental elements.

I specifically like the model of using Constraints, Requirements, Desirements, Applicability, Limitations and Alternatives as it helps indicate the optimization of the result without stating everything as a “requirement”. This model is also more open to emergent architecture elements that come from the build effort itself.

• Requirement: (Satisfied) Idempotent coding - does not assume anything about the installed / configured state of a given item. This includes elemental automation utilities like AWS CLI. This allows the code to work:
  • On a broader set of distros / editions.
  • On an AMI that has been prepared from scratch without standard AWS tooling.
  • With multiple pass processing when an instance is rebooted (already performed steps are skipped or result in no changes).
• Requirement: (Satisfied) Support both Windows and Linux (yum packaging) in all functionality.
• Requirement: (Satisfied) Handle full patching or just security patching.
• Requirement: (Satisfied) Support spot instances.
• Requirement: (Satisfied) Unique internal naming tied to stack name so that it can be deployed many times for multiple, parallel deployments.
• Requirement: (Satisfied) Least Resource Creation - only create AWS resources or do instance installs if they will be used by the specific template launch. (e.g. Providing an IAM Instance Profile Role disables the built-in role creation)
• Requirement: (Satisfied) Support terminating lifecycle hooks to trigger cleanup / deregister during scale in.
• Requirement: (Satisfied) State Based Reboot - only reboot if reboot detection code shows that it is actually needed.
• Requirement: (Satisfied) Precompile .NET bytecode after patching to ensure that new and patched .NET assemblies do not slow down production operations.
• Desirement: (Satisfied) Write code in lowest common denominator so it can be wrapped in other orchestrators. Hence this is done in CloudFormation, which can be encapsulated into other automation systems.
• Desirement: (Satisfied) Built-in scaling testing through driving synthetic CPU load across all instances in the ASG with the capability to change it on demand.
• Desirement: (Satisfied) Organize CloudFormation parameters in groups to make a more sensible interactive template deployment experience.
• Desirement: (Satisfied) Self-documentation by exposing all help information as CloudFormation parameter descriptions - increasing usability for both interactive use and automated use via integration of documentation as comments.
• Desirement: (Satisfied) Build in ASG scaling testing for learning and for production configuration validation.
• Desirement: (Satisfied) Allow IAM Instance Profile Role override with external role
• Desirement: (Satisfied) Automatic latest AWS built AMI lookup with override to peg the AMI.
• Desirement: (Satisfied) Support Gov ARNs without code modification.
• Desirement: (NOT Satisfied) It would be nice if the solution could work with a fixed patch baseline to allow full DevOps environment promotion methods using a known, version pegged set of patches.
• Limitation: The patch level is dynamic and not a fixed baseline. When scaling occurs the newest instances will have patching up to date with their spin-up date. These newer patches will not have been tested with the application.
  • Countermeasure: If you integrate automated QA testing with the provisioning of a new instance, you could catch problems with patching when they happen or by running a separate nightly build of the server tier againt the latest patches.
• Limitation: If you need to design for multiple or many reboots, you would have to do custom code to ensure userdata could pick up in the proper spot after each reboot.
  • Countermeasure: This situation is exactly what cfn-init is for, if you have not previously used it, you can read up on how to implement it within the pattern in this post.
• Applicability: If you already release a per-ASG AMI for your own reasons (usually speed of scaling), then simply ensuring that AMI takes into account your desired patching frequency is a better solution. You could shorten your AMI release cycle to something like monthly so that satisfactory patching happens as part of the existing release process. This has the side benefit of version pegging your patching level and allowing it to be part of your development and automated QA and be ensured that production runs on a tested patch level.
  • Alternative: If you have an existing long AMI release cycle (greater than 6 months), you could combine it with the dynamic patching solution offered here to keep the cycle long (to keep the cost and logistics of managing old AMIs to a minimum if that is a high priority).
• Alternative: Critical Vulnerability Response If you have an urgent enough patching scenario, you may wish to temporarily use this pattern to do dynamic patching when you do not normally support it.
• Limitation: This demo template relies on the default VPC security group being added to the instances and on it having default settings which allow internet access. If you have the default VPC security group nulled out (a great security practice!) or other networking configuration that limits internet access, you will need to update the template so that it has outbound internet access in your environment

We have built Gitlab CI Runner deployment automation to supersede our previous deployment automation for a pseudo-high availability Jenkins configuration similar to the approach of CloudBees.

We have put together a guide for developers to consider Gitlab CI whenever they have the opportunity to reconsider their CI.

These activities have given insight into the many ways which Gitlab CI differs from legacy solutions that predate DevOps and Cloud development.

This conference session will focus on an analysis of how and why Gitlab CI enables a stable, scalable, highly available, secure and easy to provision CI solution for our developers. It will contrast these capabilities to our old standby solution, Jenkins.

Session Link: https://gitlabcommit2019brooklyn.sched.com/event/TgXn/never-ask-a-butler-to-do-a-robots-job#.

I’ve spoken at a lot of conferences and I’ve experienced a good number of the many ways that conferences can fail to deliver.

GitLab Commit had three major things working against it that made me wonder what the experience would be like. First of all it was a first time conference for GitLab. Secondarily it was only one day. In my experience, many one day conferences are thinly veiled marketing events with shallow, slapped together sessions. And finally, session slots were 30 minutes - unless it is a 5 minute lightning session, I’ve never seen a session length that short.

My expectations weren’t high, but I was delighted to be wrong on all counts…

Pssst: You can still catch GitLab Commit in London on October 9th or San Franciso on January 14th.

My first hint that this was not going to be a slapped together event was what I experienced in the speaker pipeline after being accepted to speak.

The GitLab conference team had a clear lead-in project plan, a slide template and scheduled a couple practice run throughs with speakers. They also offered opportunities for interviews with DevOps.com and to publish articles at a variety of top DevOps news sites. While it was confidence inspiring to see that the details and marketing were being handled very well, the real proof came at the conference.

The evening before speaker dinner was held at Bouley Test Kitchen - it is the laboratory of Chef David Bouley. I never thought being a subject in a laboratory experiment could be so awesome. Six scrumptious small plates were served in the beautifully intimate and eclectic dining space. However, the most impactful thing was the open welcoming intimacy that the GitLab folks brought with them. From what I hear it is not hard work for them to create such social climate because it is just part of their culture.

Some conferences have real ethos to them - an energy or vibe that can be felt - but generally they have 3 or more days to get that vibe up to full volume. Taking time to wind up is not how GitLab does vibe - the Commit conference hit the ground running.

There were warm welcomes everywhere and the same positive energy I felt from the speaker treatment was available in abundance for every attendee.

My concern about session length was unfounded - speakers (including myself) had to be vigorous and on point with a clear, value focused message. So the length seemed to help create conciseness in what was delivered.

Another interesting thing was the subtle and tasteful presence of the branding everywhere. The conference was spread across three hotels, a coffee shop and a night club / bowling alley. In each venue the GitLab brand was worked into all kinds of small details and in a way that complimented the existing design features of each venue. In replacement wall posters, on mirrors, as lighting, painted on the streets, as lit up drink stir sticks and many more places. However, instead of screaming at you in neon blasts - it was blended into whatever environment you found yourself in. The conference actually blended itself into Brooklyn’s look and feel so as to feel native to the town.

Combined with the warm positive energy of the staff and presenters, the approach of integrating into Brooklyn’s vibe and amplifying it toward GitLabs brand, gave GitLab Commit an instant, full volume vibe right from the beginning.

I had the pleasure of running into the author of this approach, Emily Kyle. She summed up the uniquely engaging approach of the conference as a “Neighborhood Takeover”. And that it was - but it was not a hostile take over - rather it was “blend in and amp up the vibe”.

GitLab Commit really ended up being The Little Conference that Could for me - thanks for a refreshing and energetic event GitLab!

Just In: GitLab published a summary video that does a good job at capturing some of their conference vibe: https://www.youtube.com/watch?v=hi2D0Se_VnA

You can still catch GitLab Commit in London on October 9th or San Franciso on January 14th.

Never Hire a Butler To Do a Robot’s Job: https://www.youtube.com/watch?v=UZBjuw02gUc

Interview with Alan Shimel of Digital Anarchist and DevOps.com (Click the play icon after the page loads): https://vimeo.com/360176633#t=57m54s

Original Session Abstract: https://gitlabcommit2019brooklyn.sched.com/event/TgXn/never-ask-a-butler-to-do-a-robots-job#.

One of the main types of IaC my team builds is deployment automation for DevOps agents that are designed to run on any of the 10’s of thousands of instances at the company - agents for things like vulnerability scanning, malware scanning, log aggregation and monitoring. Generally these agents are wiring up to an internal or external cloud tenant environment for reporting and/or administration.

Everyday at my job I learn of a new environment I’ve never heard of before that someone is trying to run my team’s code in. Frequently the environment setup is at fault when these DevOps agents error out on their tenant registration calls. After way too many escalations that resulted in the discovery that the environment is at fault - I decided we need to preflight check the tenant URLs we would need to connect to and report failures in logging so that tooling users could easily distinguish when their environment was not allowing endpoint access.

Another common case for endpoint verification is when code depends on public or external package management endpoints for things like yum or chocolatey packages. However, the approach is solid for endpoints of all type whether public or private, local or remote.

If you take a look at a lot of your automation code it may make fundamental assumptions about available endpoints and if it will run in environments that are out of your control, endpoint connectivity validation will save you boatloads of support time :)

Mission Impossible Priority - while Mission Impossible Coding applies to anything that requires ruggedness, in toolmaking the return on investment is much higher since, by definition, tools are run in a huge variety of environments they were not tested for in their development cycle.

Constructed Simplicity (Conciseness) is The Tip of An Iceberg

While epitomized by the Blaise Pascal quote “I have made this longer than usual because I have not had the time to make it shorter.”, I find that the process of creating concise, simple observations and solutions is complicated and sometimes complex.

Creating concise designs and solutions is a deep passion of mine. However, there is a frustration that is quick on the heals of creating something that is concise. In my line of work, at some point, you usually have to justify your final work. That is the point at which the rest of the iceberg of thinking that backs the concise tip comes to the fore. Frequently the reaction is that it can’t possibly be that involved or that you are spinning up reasons on the fly to simply bolster an idea or solution that was arrived at haphazardly.

Why bother addressing this sentiment? Because concise, mission impossible style, solutions can easily be criticized as lacking sophistication in their implementation. But much like a Mark Twain quote - that ruddy external appearance belies a hard wrought balance of many interrelated tradeoffs to get to a simple solution.

Said another way, solutions that are earnestly designed for conciseness can be rewarded by a perception of being the opposite of what they are - simplistic, backed with little or no thought.

This Mission Impossible Coding series exposes the submerged methodological icebergs below the waterline of the visible and concise solutions it attempts to arrive at.

Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

The following list demonstrates the Architectural thrust of the solution. This approach is intended to be pure to simplicity of operation and maintenance, rather than purity of a language or framework or development methodology. It is also intended to have the least possible dependencies. It’s an approach I call “Mission Impossible Coding” because it enables the code to get it’s job done no matter what.

• Requirement: (Satisfied) Leverage TCP connect for end point validation, not ping, because:
  • It should always be possible while ping and/or UDP are frequently blocked.
  • It verifies end-to-end service connectivity right to an individual port.
• Requirement: (Satisfied) Allow input parameters to be passed as a simple string - this avoid problems when the parameter must be passed through a complex stack of languages because we aren’t trying to pass a complex data type.
• Requirement: (Satisfied) Allow multiple endpoints to be checked with one parameter.
• Requirement: (Satisfied) Keep the same data type and formatting for the parameter so that it can be passed to either Windows or Linux without transformations.
• Requirement: (Satisfied) Always informationally log what parameters are being processed before attempting to use them - this helps debugging when parameters are unwittingly being overridden or if they are malformed by a handoff somewhere above in the stack.
• Desirement: (Satisfied) Use similar code structure for Bash and PowerShell to facilitate using the scripts as a cross-language learning aid.
• Requirement: (Satisfied) Use methods that have the broadest shell version and os version support.
  • Requirement: (Satisfied) Use methods that do not require code or utilities to be installed - more important in the face of container-minimalized OS configurations.
  • Desirement: (Satisfied) Use an argument format that does not require a ton of additional code to parse in Bash (hence this solution uses a space for key value pair delimiter).
  • Serendipity: (Discovered) PowerShell version can run on PowerShell Core in native Linux, Bash version can run in Windows Services for Linux (WSL) on Windows.
• Constraint: (Satisfied) Avoid using language native function definitions to allow implementers to use their own coding styles or standards for where the loop should be and whether a function should emit informational messaging and other details.

Expansion on Architectural Heuristics

Mission Impossible Pattern Philosophy

Mission Impossible Code samples are intended to be both 1) usable directly for production use and 2) a top notch pattern to use for your own innovation. More details on why are in the post Back to Basics: Testable Reference Pattern Manifesto (With Testable Sample Code)

No Ping Because You Know Ping

The need to avoid reliance on ping as a connectivity verification tool is relatively obvious as it (and UDP traffic) is understandably blocked in so many circumstances. However, doing a TCP connect attempt also verifies the entire end-to-end conversation right down the specific required ports and includes verification that a return conversation is possible.

Reduced or Eliminated Runtime Requirements

Common approaches to the problem of verifing connectivity on Linux include using nc or telnet - both of which are not present on many minimalistic distros. On Windows PowerShell 4.x and later Test-NetConnection provides TCP connectivity testing - but is not available on PowerShell Core nor older versions of PowerShell.

All of these are avoided by the solution code in this article so that it can run in the broadest number of contexts. As an unintended positive consequent, it turns out that the PowerShell code works on Windows and Linux. The Bash version also works on Windows Services for Linux running under Windows.

Mission Impossible Priority - The unintended positive consequences of reducing or eliminating dependencies are frequent and delightful, like biting into a jelly donut for the first time ;) Dependency reduction / elimination is usually worth it.

First Class Design Priority: Parameters Get Passed Through Complex, Multi-layer Stacks With Value Overrides Allowed At Many Levels

It is important to consider that shell code must be called by something else to run - sometimes by a huge, multi-layered stack of “something else”. In complex automation stacks, each hand-off between layers represents risks for data types that are more complex than a string.

This code could have been built to take a complex data type like a hashtable or array. In my experience this generally leads to massive rabbit holes of effort to pass the data between layers of automation infrastructure. Since each language handles the syntax of complex data types differently, such data may need to be escaped or reformatted between layers due to parsing constraints. Many times specific layers don’t have complex data types or it takes special transformations to pass into the next layer.

To further frustrate the efforts, it can be very difficult to get debugging visibility to the hand-off between these processing environments. On top of this lack of visibility, most parsers don’t give direct errors on the problem. In fact, many times your parameter, now malformed, is not detected until you try to use it.

The string data type is the most elemental and universal. By using a simple, structured string - the pain of complex type passing can be avoided and parsing is handled by the shell code once the data is received into the shell environment. Not only does this allow passing through many layers of automation with ease, it allows Windows and Linux to share the exact same format. Which in my case, allows the same argument to be seamlessly passed to the Windows or Linux branch of the code at the appropriate point in the many layered automation stack.

Mission Impossible Priority - how many times have you seen yourself or others stay idealistically fixated on using a data type to the point of wasting many hours. Here is a key place where a dedication to “Mission Impossible Coding” dictates a pragmatic override of idealism to what will consistently work in the broadest number of scenarios. Idealism is a secondary to pragmatism.

By the way, the format I have chosen here may not be friendly to your specific stack of parameter handoffs - but you can see that it uses a space to delimit key value pairs and equals sign to delimit key from value - feel free to adjust those in a way that is compatible with your stack. In fact, if anyone has devised universal delimiters that pass through most known layers without incident - I’d be interested on a comment on this article.

Logging From A Toolmakers Perspective

A healthy orientation to verbose informational logging is a toolmakers friend as it encourages development users to self-diagnose and self-resolve problems they would otherwise reach out for support on.

A final point to consider with parameters is to always 1) informationally log them 2) before attempting to use them. Informationally means don’t only log errors - this handles use cases where the bottom level automation is receiving working parameters, but they are not the correct.

The complex stacks our parameters pass through frequently allow parameter overrides at many of these levels - informationally logging them allows a super quick find of an unwitting parameter override. Once again, more critical if your automation is tooling used by others since you don’t have control over the parameter stack. The reason for logging before using them is so that any error handling that might happen does not negate your ability to report the parameters in use at the time.

Mission Impossible Priority informational logging of parameters before API calls: 1. Always communicate verbosely with your support team, 2. When you are going to die, write a note who dun it to you.

Built-in List Processing From The Start

This code was built from day 1 with processing a list in mind. This design heuristic is stolen directly from PowerShell where CMDLet design makes it exceedingly simple to process lists of things that are pipelined in - so simple, there isn’t much reason not to do it. No matter what language you prefer, how many times have you had to refactor a solution to process a list when you originally built it for a single value. Hard to list them all isn’t it?

Timeout is Required

Initially I didn’t bother with timeout functionality in the PowerShell version because the default was bearable. However, when it was run on Azure Cloud Shell - it was unbearable. A strong commitment to pre-mission testing was the only reason this came to the fore.

Mission Impossible Priority - relentless pre-mission testing of all alternatives and contingencies is standard operating behavior.

Mission Impossible Code Bonus: Minimal, Universal Logging

Logging is fundamental to Mission Impossible Coding. I also have Bash and PowerShell code designed with the same eye to minimal, universal reuse. You can implement it with this code by retrieving it from here: MICode-MinimalUniversal-Logging

The Code

Here is the Bash - last url will purposely fail.

urlportpairlist="outlook.com=80 google.com=80 test.com=442"
failurecount=0
for urlportpair in $urlportpairlist; do
  set -- $(echo $urlportpair | tr '=' ' ') ; url=$1 ; port=$2
  echo "TCP Test of $url on $port"
  timeout 3 bash -c "cat < /dev/null > /dev/tcp/$url/$port"
  if [ "$?" -ne 0 ]; then
    echo "  Connection to $url on port $port failed"
    ((failurecount++))
  else
    echo "  Connection to $url on port $port succeeded"
  fi
done

if [ $failurecount -gt 0 ]; then
 echo "$failurecount tcp connect tests failed."
 exit $failurecount
fi

Here is the PowerShell - last url will purposely fail.

$UrlPortPairList="outlook.com=80 google.com=80 test.com=442"
$FailureCount=0 ; $ConnectTimeoutMS = '3000'
foreach ($UrlPortPair in $UrlPortPairList.split(' '))
{
  $array=$UrlPortPair.split('='); $url=$array[0]; $port=$array[1]
  write-host "TCP Test of $url on $port"
  $ErrorActionPreference = 'SilentlyContinue'
  $conntest = (new-object net.sockets.tcpclient).BeginConnect($url,$port,$null,$null)
  $conntestwait = $conntest.AsyncWaitHandle.WaitOne($ConnectTimeoutMS,$False)
  if (!$conntestwait)
  { write-host "  Connection to $url on port $port failed"
    $conntest.close()
    $FailureCount++
  }
  else
  { write-host "  Connection to $url on port $port succeeded" }
}
If ($FailureCount -gt 0)
{ write-host "$FailureCount tcp connect tests failed."
  Exit $FailureCount
}

Tested On

PowerShell Version (Windows and Linux)

• PowerShell 5.1 on Windows 10
• PowerShell Core 7.0.0.2 on Windows 10
• PowerShell Core 6.2.1 on Amazon Linux 2 in Docker on Windows 10
• PowerShell Core 6.2.2 on LinuxMint 19.1 On A Laptop
• PowerShell on Azure CloudShell

Bash Version (Windows and Linux)

• Ubuntu 18.04.1 LTS in Windows Services for Linux (WSL) on Windows 10
• Amazon Linux 2 in Docker on Windows 10
• Bash on LinuxMint 19.1 On A Laptop
• Bash on Azure CloudShell

Code For This Article

MICode-MinimalUniversal-TCPConnectPreflightCheck