DevOps CheckLists: Why Human-Processed Source Code Belongs in Git (Part 1)

It is easy to think of code as only a computer thing, but in truth, humans have been codifying all kinds of knowledge for centuries. There is one type of human knowledge that is more like computer code than anything else: and that is process execution knowledge.

We are all familiar with the concept of a computer executing code — it takes a structured instructions and executes them in order to produce a desired end result. Computers are very good at this meticulous duty and perform instructions flawlessly many times over.

So what would the equivalent be in the human realm? How about a detailed set of ordered steps that must be completed to achieve a desired result? That seems like an apt description of procedural checklists.

The book “The Checklist Manifesto” paints a picture of how checklist usage creates a dramatic impact on quality outcomes by achieving consistency in meticulous and repetitive human-performed tasks. Checklists literally save lives in many high stakes circumstances like flying airplanes and performing surgery.

GitOps is a commitment to ensure everything needed for success in building and operating software is stored in Git. In fact, Git ultimately serves as the single source of truth. However, most of the time we secretly limit that “everything” to “everything the computer performs” and leave out the things that humans do to ensure the integrity of the software.

While a true commitment to the “everything in Git” principle would encourage the storage of checklists that humans perform in Git, there are some much deeper connections between the human code and the computer code that eventually automates these procedures.

The real focus here is operational checklists for the success of software (although we’ll discuss a lot about checklists in general). Think of them as build, deployment or testing steps.

The Genesis of All Process Oriented Work

Process-oriented computer work has its roots in work done by humans. We want to be relieved of the tedium of precise, ensured execution of extremely detailed, order-sensitive steps. If that work is done in a repeating cycle, we have all the more motivation to be rid of and to quality ensure it by giving the work to machines that don’t experience mental drift due to tedium.

The Best Process Discovery Mechanism

I have had periods in my career where overnight air travel was frequent. Since I have a busy mind, the possibility of forgetting to bring something was very real and caused a lot of stress. Overnight air travel involves several “point of no return” milestones because at a certain point you no longer have available time to double back to your home to grab an item and later, you can no longer double back to your vehicle to grab something either.

The power of a master checklist to solve this has been amazing. Over the years of keeping this checklist, I’ve noticed something else: that the process of creating and executing the checklist causes things to come to mind that need to be in it (or removed from it).

Sometimes these are completely new types of things that I hadn’t previously thought of putting on the checklist. This checklist is now ordered by T-minus transitions — the same way spacecraft launch checklists are tracked. Things like “Before Leaving the House” and “Last Minute Grabs From the Car.” It also has conditionals like If International Travel, If Empty House.

Brain Dumping and Mental Associations

It seems when we create a concrete checklist from our tacit knowledge of a process, it activates our minds to find related items. This is a natural part of human cognition, but is especially helpful with regard to discovering edge cases and conditional steps. The long term maintenance of a master checklist also continues to add value in this area as there is someplace to put new ideas about what ought to be done to make a checklist more effective at assuring its proper execution.

Tacit Process Knowledge Made Explicit

When we build software to take the place of human process-driven work, the human checklist is an invaluable source of information. Even if the checklist has to be made specifically for a software build, it provides a full extraction of the tacit knowledge of those performing the processes because it can be validated by someone with no tacit knowledge — which is exactly what computer code has to do because it does not have tacit knowledge either. Tacit process knowledge is the knowhow that experts have a hard time explaining to you because it has become so reflexive to them they do not notice that it is an important part of process execution for those who are unfamiliar.

Development Backlog Refinement

The building and refining of checklists really represents a somewhat automatic refinement of process automation requirements. Rather than trying to empirically discover what should be done while coding how to do it — discovery and coding can be separated. This helps when there are different individuals doing each activity, but it also helps when the same individual does the activity because separating the flow states of “coding” and of “focused checklist execution” yields the top-notch awareness that improves the quality of outcomes for both activities.

Discerning Low and Negative Automation ROI

While the “Automate All The Things” meme is funny, it is also a commentary on a philosophical position taken too far. Automation is about efficiency, which in turn is about return on investment. Several brave writers and bloggers have published guidelines on situations where the ROI is not there for automation. These tend to be the areas where human judgment and/or strong variability between executions are an inherent characteristic of a process. Checklists help discern where automation ROI may be low because the attempt to create a checklist with a lot of judgment calls helps articulate whether the code required to make the same judgments is possible and cost-effective. Another indicator is that a checklist needs new custom steps for every execution. These characteristics indicate there would be more time spent maintaining the code of the automation than is involved in humans performing the tasks — perhaps by orders of magnitude. The process of attempting to automate can also weed out which sub-processes which naturally have a low automation ROI, while also automating anything that is a candidate for automation. Discerning low or negative ROI can be done during the overall automation discovery process — but the existence of low/negative ROI must be believed in and filtered for to avoid the “automate it no matter the ROI” trap.

Human Code Update, Review and Adoption

Just as computer code benefits from collaborative-maintenance updates, so human code — in the form of master checklists — can also benefit from this collaborative approach — whether or not it can ever be boiled down to automation or not. Collaborative coding is a main feature of Git and gaining the benefit is as simple as ensuring checklists are stored in Git, in a text format so that Git can perform human-readable diffs on changes. Essentially this naturally leads choosing Markdown as the document format.

Human Code Auditability

Since humans have a challenge performing meticulous, repetitive activities with high fidelity, we tend to want them to provide proof that they performed the actions as prescribed or describe what alternative action was taken. By storing completed checklists in Git, we can facilitate auditability. As discussed later, we can determine what type of “DONE” marking is used in order to improve the auditability — such as whether to include a date time stamp in the DONE marking of each step.

Automation and Checklists Go Better Together

Some DevOps Engineers might experience frustration over spending effort on codifying checklists rather than directly creating computer code to do those activities. The above list shows very valid reasons why it makes sense to acknowledge and purposely incorporate checklists into the process of initial and ongoing maintenance of automation. The risk that something might not be automated because it is too well documented as a human procedure should not lead us to ignore the fact that non-codified human procedures are a large scale risk to the integrity of the full DevOps lifecycle of a software solution.

Live Checklist Status Sharing

During a change event, it is very handy to understand where everything is in the process — especially if multiple checklists must be performed simultaneously. While some of the SaaS checklist solutions offer live sharing as a primary value, Git-stored lists can come close if the checklist being watched is frequently pushed to a git server and individuals viewing it refresh their view. You could even set up a looping “git commit and push” script on the workstation executing the checklist.

Code Is Code Is Code and Should Be Stored in Git

It’s common to think of “code” as being computer code — but human code fits the model in the form of checklists, with very detailed, order-specific and interdependent execution instructions. When we record human code in a way that leads to effective execution — we naturally arrive at the concept of a checklist. As we’ve discussed human code has many similarities to computer code — including how it benefits from the visibility and collaborative capabilities Git offers.

When human code is critical to the development, build, testing and operation of a software stack — it fits the definitions of GitOps — the single source of truth for success in managing a software stack.

Well-written and -maintained human code is the best starting point for creating computer code.

In a follow-up post we will look at two free, open source and multiplatform tools that make the creation and execution of markdown-based checklists much easier. It will also include configuration code that gets you started with easy “done marking” to keep the stress of checklist execution as low as possible and consistent between team members.