Automated Collection of Diagnostic System Information From Systems You Don't Own
Over time I’ve found there are many reasons why I may be asked to debug systems I do not have convenient access to.
It is painstaking to get lists of diagnostic information in circumstances such as these because it requires a lot of information relaying and lag times.
In the past, I have done a survey of the available options for systems information utilities to help get a dump of a system’s configuration information.
This post discusses the key selection criteria for such a utility and provides automation code (including oneliners) for collecting the diagnostics information with minimal relaying of instructions.
Escalation Toolkit
I plan to publish more of these scripts and tools designed to ease the burden of quickly getting the right tools onto machines that you need to troubleshoot. Subscribe to this blog and check the escalation-toolkit tag for related posts.
Root Cause Analysis (RCA) - More Than Terminology
In IT, the term “Root Cause Analysis” (RCA), is thrown around as a synonym for debugging or troubleshooting. In reality there are specific disciplines around how the troubleshooting should be approached. Escalations usually have to do with problems that are hard to understand.
They should also be an inflection point at which time the troubleshooting turns from tactical to strategic.
Tactical Troubleshooting is when many solution attempts for common probable causes are simply tried on a system. They can be very productive if the system is suffering from one of the top 10 known problems in a given environment.
However, once these are tried, troubleshooting should shift to a strategic mode. The strategic mode is characterized by collecting and formally recording more data. One thing this data is generally used for is “comparing to a known good scenario”.
I have a personal theory that when our troubleshooting tools are not easy to deploy and use, that there is natural friction in collecting all the data that would be helpful for diagnosis. When such data is collected it not only helps the current technician compare between systems and compare the same system over time (if inventory history is available or change dates are attached to the inventory) - but it also helps collaborate - whether collaborating concurrently or when escalating or transferring the problem to another technician.
System inventories are usually rich with problem determination data, as they frequently reveal the differences between two supposedly identical systems.
The Scenarios
Generally, when I find the need to do inventories, a problem has been escalated to me and there is another technician who has access to the system that I am working with - but direct access to the system by myself is not possible or too challenging to configure or authorize for a troubleshooting session or two.
It is also helpful to gather machine inventory output:
- to get a feel for the target system before an interactive screenshare debugging session is scheduled
- to compare to a known working system to detect significant differences
- to record hard facts of inventory in a ticket for others to help with diagnosis - concurrently or when the ticket has to be passed to another technician.
Key Tool Selection Criteria
- Free - so that licensing is not a problem.
- Can Export Data - so that it can be shared without live system access AND so that it can be stored in documentation or support tickets and for comparing to known good systems.
- Collects Appropriate Data - for me this is at least software inventory and patch inventory. It is surprising how many utilities collect tons of hardware data, but either leave out software or patches.
- Does Not Require Installation (Portable) - so that it does not soil the system being inventoried - especially not upgrading shared runtimes (C++, .NET, etc) or other components used by the software on the system being inventoried.
- Can Be Automated Via a CLI - so that I can send commands that can simply be run rather than long explanations of how to operate a GUI tool.
- Can Filter Out Secret Data - so that the output can be safely shared across companies.
ESET SysInspector
ESET SysInspector meets all these requirements and desirements. It also has the enviable quality of not being zipped up - so it is easy to pull onto any system.
ESET collects all the data I need for most situations. The only possible improvement would be more data on automatically starting applications. But we can augment with autoruns for that.
Example Code
As per the [Testable Reference Pattern Manifesto] (https://MissionImpossibleCode.io/post/back-to-basics-testable-reference-pattern-manifesto-with-testable-sample-code/) all of the below code has been tested. It is also available in a repository to help avoid problems when copying and pasting from web pages: [MissionImpossibleCode] (https://gitlab.com/missionimpossiblecode/MissionImpossibleCode)
Download and Run ESET SysInspector (with privacy filtering and export to a zip file)
#Automatically downloads Eset, runs it with privacy screening and exports to a zip file.
Invoke-WebRequest -Uri 'http://download.eset.com/download/sysinspector/64/ENU/SysInspector.exe' -outfile "$env:public\SysInspector.exe"
cd $env:PUBLIC
$Filename = "$env:PUBLIC\SysInspector-$env:Computername-$(Get-date -format 'yyMMdd-hhmmss').zip"
Write-Host "Starting ESET SysInspector - it is not unusual for it to run up to 10 minutes as it collects a lot of data"
$ProcessHandle = Start-Process "$env:PUBLIC\sysinspector.exe" -ArgumentList "/gen=$Filename /silent /privacy /zip" -Passthru
Do {
++$ElapsedTime
Start-Sleep -Seconds 60
Write-Host "Been waiting $ElapsedTime Minutes..."
} until ($ProcessHandle.HasExited -eq $True)
Write-Host "Please send this file to the requester: `"$Filename`""
Write-Host "You can also run `"$env:PUBLIC\sysinspector.exe`" to open the file and examine what was collected."
This oneliner executes the above code straight from github:
Invoke-Expression (Invoke-Webrequest -uri 'https://raw.githubusercontent.com/DarwinJS/CloudyWindowsAutomationCode/master/CollectAndPackageSystemInfo.ps1')
Download and Run (autoruns and export to a .arn file)
#Download and automatically run autoruns.exe to generate an exported .arn file.
If (![bool]([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator"))
{$erroractionpreference = 'Stop' ; Throw "You must be an administrator to run this!"}
Invoke-WebRequest -Uri 'http://live.sysinternals.com/autoruns.exe' -outfile "$env:public\autoruns.exe"
cd $env:PUBLIC
$Filename = "$env:PUBLIC\Autoruns-$env:Computername-$(Get-date -format 'yyMMdd-hhmmss').arn"
Start-Process "$env:public\autoruns.exe" -ArgumentList "-e -a $Filename" -wait
Write-Host "Please send this file to the requester: `"$Filename`""
Write-Host "You can also run `"$env:PUBLIC\autoruns.exe`" and use File => Open to open the file and examine what was collected."
This oneliner executes the above code straight from github:
Invoke-Expression (Invoke-Webrequest -uri 'https://raw.githubusercontent.com/DarwinJS/CloudyWindowsAutomationCode/master/CollectAndPackageAutoruns.ps1')
Updated Code In This Article
The above code’s primary home is in the following repository - where it might be improved or fixed compared to the code posted in this article. It is also safer to use the code from the repo rather than copy and paste from this post: https://gitlab.com/missionimpossiblecode/MissionImpossibleCode