While noodling uses for the Ultimate AWS ASG Kickstart and Lab Kit I realized it could use a couple new features and improvements.
The first is to have instances self-tag themselves as to whether they are a spot or on-demand instance. When supporting a mixed instances policy, the implementation requires a little more thought.
A second is the ever common need for a bucket that the ASG instances have access to - whether for AWS SSM results collection or inventory, for deployment artifacts to update instances, access to data or many other uses.
While a relatively small change, the title has also been updated to “The Ultimate AWS ASG Kickstart and Lab Kit” to indicate it is appropriate for experimentation and also as the foundation for a deployable configuration.
I was able unable to avoid the temptation to sneak in a couple other improvements as well.
TL;DR Feature Summary
Skip learning new ideas and go straight to the CHANGELOG.md If you missed the original article it is here: The Ultimate AWS AutoScaling Group ASG Lab Kit
New Features Checklist
- Instances tag themselves as spot or on-demand.
- S3 Bucket Setup
- Managed Permissions Instead of In-line
- Optionally provide a keypair name for remote access.
- Use Cloudformation “Rules:” for cross parameter validation.
Features in Detail (Ideas)
Instances Tag Themselves as Spot or On-Demand:
One of the downstream uses I intend for this Kit is to build a GitLab CI Runner on it and I realized that ephemeral compute is good for some CI workloads which are more interruptible - like mass automated testing - but not good for others where the impact of interruption might be much more significant - like long running deployment processes. By allowing instances to self-identify and be selectable by CI engineers, they can make their own decisions about what CI workloads to run on ephemeral compute.
The resultant Instance tag is either COMPUTETYPE=SPOT or COMPUTETYPE=ONDEMAND. The variable $COMPUTETYPE is available throughout the userdata script including if you are using the template to pull in your own userdata code. By leveraging this variable you can surface this data to other systems you may be installing on the instance. For instance, I will be adding a GitLab CI Runner tag with this data.
An interesting detail here was that attempting to tag by propagating from the ASG LaunchTemplate won’t work for a mixed instances policy - when the LaunchTemplate might launch a mix of spot and on-demand, it is important that the instance self-detect and self-tag.
Many internet searches directed me to use the aws call “describe-spot-instance-requests”, however, in the spirit of Least Privilege and Least Config - I dug a little deeper to find that it could be done purely with “describe-instances”. Since this template already required describe instance permissions to handle ASG Lifecycle hooks, using “describe-instances” meant that I could use familiar code and not have to add permissions
S3 Bucket Setup
As with many other resources this template can either an auto-create a bucket for you, or you can override bucket name with one that exists. In either case, Permissions are applied to the instance profile.
Managed Permissions Instead of Inline
Previously the template created permission using inline IAM Policies attached to the created instance role. This meant they could not easily be attached to existing roles. By creating IAM Managed Policies, they are easily attached to any existing role. They are also created even if you do not have the template auto-create the IAM Instance Profile Role.
Use CloudFormation “Rules” for cross parameter validation
It is a common mistake (at least for me) to choose “Windows” as the platform, but then forget to update the SSM parameter path for image lookup to point to a Windows image. Unfortunately you don’t discover this until after the machine starts. Now the template prevents execution when your parameters are in this state. It also serves as a working example of CloudFormation rules, which hard to come by - especially in YAML.