Think about how babies initially learn - purely by observing patterns - but more specifically “actual working patterns”. They watch people actually walking, actually talking and actually eating in order to learn these activities - and their shiny new brain is really, really good at it.
Recently I published a PluralSight course on implementing the Center for Internet Security (CIS) AWS Foundations Benchmark security standard. As a part of that course I wanted to provide a working script that could make a clean, test AWS account compliant with the benchmark (an “actual working pattern”) I am writing this post to release that code as open source, as well as to relate some lessons learned during the journey of building it.
CIS Benchmarks: Pragmatic To The Point of Providing Code
CIS Benchmarks are unique among security standards because they take an extremely pragmatic approach to defining security. Instead of doing it in a general, descriptive way they stipulate the technology specific settings and demonstrate exactly how to update them to be compliant with the benchmark. The AWS benchmarks go so far as to include lines of code for doing an audit and for remediating the settings if they are non-compliant. If extracted from the document into a script, the complete code is intended to provide a sample, or reference specification, for how to make an AWS account compliant. Additionally, they could be said to be “test driven” because the recommendations discus and give the actual code for the audit procedure first.
An interesting aspect of this pragmatism is that it is shared by some of our most beloved technology development methodologies such as Agile, DevOps, Automation and Continuous Integration. This commonality has a two part synergy of making CIS benchmarks suitable for: [a] keeping up with the rate of change driving by these methodologies and [b] being implementable using these methodologies.
A Practical Journey
I thought that it would be a useful exercise to actually extract and compile these commands into a working script that could configure an entire AWS account to be compliant - “should be easy, all the code is right here” I thought. However, it turned out it was not that straight forward. To ensure that all dependencies were handled and that everything was processed in a sensible sequence, I had to add a lot of code. Additionally, the reference code tended to be overly verbose - for instance it contained 8 different pieces of code for setting the AWS password policy, when most of the password policy elements can be set with one line of code.
But more than that happened - as I assembled a working code base and iterated over configuring an actual compliant account, I realized it helped me understand what the benchmark was doing, for instance:
- The benchmark was actually directing me to build a scalable Security Event and Incident Management (SIEM) system using the componentry of AWS.
- I could identify process or knowledge gaps - for instance, level 2 of the benchmark specifies that AWS Security Groups should be “Least Privilege” - but it is not in the scope of the document to discuss the practical engineering required to devise truly least privilege Security Groups (in that case I created an entire course module to cover how to do that engineering.)
- I could easily test whether the benchmark code worked properly and make adjustments.
- I was able to discover outright errors with the code provided in the benchmark and readily identify when it was incomplete for the stated task.
- I was able to identify improved approaches compared to the original code provided.
- I could easily add my own checks that the benchmark did not specify.
- I could more easily identify which parts of the benchmark would have a high impact to an AWS account that was already in use.
Testable Reference Pattern Mini-Manifesto
After this effort I started to recognize that in technology we deal with a lot (a whole lot) of “reference code” - we sometimes call them “samples”. And almost all of these samples suffer from a similar problem - whether you find them in a “hands on book” or posted in a forum.
That problem is “the code is not tested” - someone pseudo-coded them up and when it comes to running them, they don’t tend to work.
They hold out the false promise of being a fast path to understanding and productivity and frequently they are the opposite - they can actually drain away valuable time.
This common scenario seems out of place as IT has solved the “the code is not tested” problem for regular application code some time ago. This is done by the discipline of built-in unit testing and automated QA testing as an integral part of Continuous Integration automation.
Consistently re-suffering from the havoc of a problem that has already been solved many times over, is like having someone run a cheese grater up and down the left side of my head. Since this is an unpleasant experience (and I’m almost out of sharp cheese graters), I am adopting a personal position of working examples - whether it’s just my personal snippet library or examples I provide to others.
I actually have another example that pre-dates my experience with the CIS AWS Benchmark. In a previous position, I joined a team of software deployment automators and was soon tasked with getting the teams tooling and skills onboarded to PowerShell. The team was very adept at handling their daily tasks by using an existing non-PowerShell framework for doing this work and, as with many operations teams, there was a lot of pressure to keep productivity high while moving the tool chain and skill set forward.
I had a watershed moment when I realized that I could best influence the team skills by first migrating the existing framework’s primary reference pattern library forward to PowerShell. This is because they could be productive quickly while simultaneously learning PowerShell gradually. After working on this project for a while I realized a further catalyst to adoption would be to fix the challenges and add desired improvements of the old framework in the new. Then team members would be stepping up to more capabilities as well as upgrading their skills at the same time.
This is when I really had the first aha about testable reference patterns - the old frameworks “starter template” (or default scaffolding) was NOT a working reference pattern - it had to be tweaked to work - as a new person to the team you had to bump along the road learning all the parts that were missing from the “granddaddy template” to get your first, working rendition.
So I changed this approach by making the “starter template” work right out of the box. I kept it relatively simple, but demonstrating the value of the framework versus custom coding or the old framework.
“Testable Reference Patterns” Is Everyone’s First Ever “Learning Mode”
I feel the portion of the human mind dedicated to learning by recognizing and adopting patterns is huge. Think about how babies initially learn - purely by observing patterns - but more specifically “actual working patterns”.
They watch people actually walking, actually talking and actually eating in order to learn these activities - and their shiny new brain is really, really good at it. The closer to “actually working” the input patterns are - the higher the value to their pattern-based-learning brain.
Eventually children are introduced to non-working patterns as a learning approach - this happens when adults interject to “teach” them using “incomplete patterns” that reflect our belief in deconstructionist teaching rather than the natural world’s default of “holistic experiences”.
Do I believe there is no place for non-working pattern learning? No. However, I believe that the depth of how effective working patterns can be - and therefore how deep my commitment to them should be - is justified by the preeminence that this approach has in every individual humans learning experiences. Working patterns should have preference as the primary method whenever they can.
Its not just an interesting idea that works for some people - its an integral part of being human.
Reference Specifications That Involve Code Should be Testable
While it would be nice to have confidence that all sample code ever generated or posted was pre-tested - my main focus here has to do with “Reference Specifications”. Reference Specifications, by definition, are designed to nearly exhaustively demonstrate capabilities of the technology framework that is being supported by the specification. In the case of “Reference Specifications” the value of working, testable code is many fold.
I feel that there is a deeply fundamental reason that code reference specifications should be testable. That fundamental reason is that reference specifications exist for the purpose of being a catalyst to driving adoption of the technology framework they demonstrate. They share this fundamental aspect with “Working Samples”
Here are just some of the benefits of a Testable Reference Specification
- testable reference specifications essentially make development of the technology framework a test-driven pursuit which surfaces any presumptions of implementing a part of the functionality.
- testable reference specifications easily function as a test harness for the entire technology framework (to the degree that they demonstrated all functionality).
- testable reference specificatis allow easy onboarding to an entire technology framework as practical implementations are simultaneously readable and runnable.
- testable reference specifications facilitate adoption of enhanced functionality of a framework because it is easy to kick the tires on new stuff. An ongoing challenge with continuously improved frameworks is that their enhanced features are not as easily taken up because of the time involved in learning how it works in a practical way so that the effort to adopt new features can be easily assessed.
Objection! A Complete, Testable Reference Specification is Not Always Possible Nor Practical
I would concede that there are frameworks for which this may not be practical. However, I would never give that concession quickly because it immediately dampens the Mother of Innovation, “necessity”. So if we hold that it is a “necessity” that our reference specifications are testable, we end up coming up with innovative ways to “make it so”.
For instance:
- creating a set of “shared, core required” patterns that are leveraged by a set of much smaller, easier to maintain sub-patterns
- breaking up the reference specification into sub-units that demonstrate specific implementation patterns (whether or not there is a core set of patterns)
- have individuals or groups who maintain associated functionality maintain sub-patterns in order to share the maintenance load.
No It’s Not Just a Huge Sample Library
So the question arises, what is different than this working reference pattern concept and a just having a very exhaustive sample library?
- sample libraries do not usually intend to give full coverage to the functionality of the framework they are for.
- sample libraries can differ vastly in their applicability to the processing of learning and adopting because they are not generally contrived together as a set.
- sample libraries are not used to do automated testing of the framework.
- sample libraries are not generally kept up to date to be bug free and use the most recent framework methods and code available for the task (after all they are just samples)
- sample libraries do not share common variables throughout - an aspect of a testable reference specification that makes it much easier to map the framework concepts to minimum working code.
- authors of samples for a library do not feel committed to testing samples with each release and to maintaining their quality.
Push Hard to Make It Testable
I now work harder at thinking about how a working reference sample can be the first thing I create when explaining a coding framework or a coding framework capability or implementation pattern. Then I work backwards to the minimal additional amount of documentation required given the existence of working code that someone can examine and play with - if possible I do this as code comments.
I think a commitment to Testable Reference Patterns is a catalyst to adoption of code at many levels. I hope you now share this vision and will give it a try in the projects you work with - I feel confident it will be rewarding.
This Concept In Production
Here is a link to my version of a Testable Reference Pattern for Version 1.0 of the CIS AWS Foundations Benchmark security standard.
I have used it to configure a production account for compliance with the Foundations benchmark.
It is written in PowerShell and was tested as working on both Windows and PowerShell Core under CentOS 7.
https://gitlab.com/DarwinJS/AWSSecurity
