The Secureworks Incident Response (IR) team responds to incidents at a wide variety of client organizations, each of which have unique threat profiles, campaigns and actors targeting them. As responders, we love the challenge of being prepared to respond to vastly different incidents; however, with unfamiliarity comes challenges. Logging practices should aim to close gaps in visibility and retention that would otherwise hinder investigation and response. Yet we often see organizations approach log retention as a regulatory check-the-box exercise, resulting in missed opportunities to capture what matters most in an investigation down the line. Perhaps the greatest example of this is the tendency to overlook “what is being logged” for the sake of “how long” it is being logged.
Windows Event Logs
Since the introduction of Windows NT in 1993, Microsoft’s Windows operating system has logged system events into the Event Log format in either the System, Security, or Application event log. With the introduction of Windows Vista in 2006 this system was overhauled, allowing additional event log files, with modern systems having over 100 individual event logs. This shift has allowed for highly granular event logs that play a key role in incident response, such as tracking lateral movement and service installs, for example.
Login Events
The Security event log contains a wealth of information key to tracking lateral movement and account login activity to include source workstation and account name used. Unfortunately, by default this log has a maximum size of 20 MB before the oldest events begin to “roll over,” or get overwritten, as new events are recorded.
Losing security event log entries, especially 4624 successful login events, can make it difficult and time consuming to identify the source of a login event, preventing the determination of root cause.
PowerShell
Threat actors continue to leverage and abuse PowerShell for a wide variety of activity from creating files to credential theft involving invoke-mimikatz or similar tools. PowerShell event logs can contain a wealth of information on PowerShell executions, including when it was executed and by whom.
Even when knowing that PowerShell was executed by a known compromised account, actioning that information is next to impossible without knowing what was executed. Starting with PowerShell v5.0, Microsoft introduced the ability for script block logging that results in the full PowerShell command being logged. The problem is that it is almost never enabled. When it is enabled, it is normally just a small subset of systems within an environment. Seeing the full PowerShell execution can provide great details on the attacker’s tactics, tools and procedures (TTPs).
Windows Management Instrumentation
Windows Management Instrumentation (WMI) is an often-overlooked component of Windows operating systems that facilitates remote management of systems. Secureworks first observed WMI being abused by threat actors in 2016, and its usage has only increased since then. WMI can be utilized for a variety of malicious actions including network and system reconnaissance, malware persistence, and download of files. Without proper logging, actions taken through WMI will likely not be identified. With the introduction of Windows Vista, WMI activity can be logged to event logs, but our experience in the field during Incident Response engagements show that it rarely is.
Process Auditing
While forensic artifacts that indicate historical executions of programs (userassist, shimcache, amcache, etc.) are well understood and relied upon, they do not provide the full picture.
Occasionally during Incident Response engagements, clients will have enabled windows event logs process auditing (4628). Unfortunately, this event type can be configured in two ways, with or without command line auditing, and in most cases, it is configured without.
Without capturing the full command line key information needed for malware analysis (e.g. execution flags/password) or forensic analysis (e.g. specific script executed by WScript), it is often lost and unrecoverable.
Process creation events, including full command line, can be captured within the Security event log. This information can help recreate threat actors’ behavior to help better understand, and ultimately remediate, an incident. Note that this will increase the amount of data written to the Security event log, potentially leading to other key data being overwritten.
This data can also be captured with EDR tools or Sysinternals Sysmon.
Load Balancers
Secureworks frequently requests and analyzes network level logs while responding to incidents. Often, one of the first things identified is that log entries for externally facing resources all contain the same source IP address, that of a load balancer. While load balancers play a key role in network operations, without proper configuration logs will just see the load balancer as the "source” of the request. While load balancers maintain their own logs, attempting to reference these against, say web server logs, is often a fruitless exercise due to slight discrepancies in each system’s time. Not to mention it significantly slows down analysis.
To prevent this, the X-Forwarded-For request header can be configured to forward the true source IP address. In Secureworks’ experience, this configuration is often overlooked.
*nix Command History Files
On Linux/Unix systems, commands executed through a command interpreter are logged by default to a history file for the specific shell and user profile that executed it. This provides a large amount of valuable data as the entire command line is captured; however, the specific time the command was executed is not enabled by default. Without logging timestamps, the time a specific command is executed can only be estimated by cross referencing filesystem data for any command that would create/modify a file. In Secureworks experience it is rare for command shells, bash specifically, to have the HISTTIMEFORMAT option enabled.
Many of the concerns with log data discussed can be alleviated through centrally collecting and retaining data or via the implementation of an Endpoint Detection and Response (EDR) solution that captures full command line arguments and login events.
However, having a solution in place does not provide value if its use is not well understood by security teams. Secureworks recommends that teams be proficient in utilizing solutions to include exporting data into CSV formats. Increasing the verbosity, duration and accessibility of log data is one of the easiest ways to increase the speed and accuracy with which incident responders can do their jobs. In fact, the best way to address these capabilities is to build a response program using a proactive incident response approach that includes training, workshops and exercises geared toward the technical audiences who are critical to organizational preparedness.