Research
SecureWorks - On the Radar Newsletter - February 2008
On the Radar

Log Management Explained

Logs stem from every system on your network, detailing all of the activity on your network. This makes logs a vital source of information for security, regulatory compliance and IT troubleshooting. Unfortunately, it also means that millions of logs are produced by the devices in a typical IT environment every single day. Making sense of this large volume of log data is no simple task and it requires having the right technology, people and processes in place to manage your logs.

Log Management consists of two core processes: Log Monitoring and Log Retention. The following graphic illustrates the major steps in each process:

img

While complimentary, each process is different in both structure and purpose. Log Monitoring is an operational security process aimed at detecting incidents in real-time before they impact the confidentiality, integrity or availability of IT assets. The purpose of Log Retention is to store and archive large volumes of raw log data so that it can be easily retrieved and searched at a later time.

Log Monitoring

Several steps are involved in Log Monitoring. First, you need to collect the logs being generated by the systems on your network. Having “situational awareness” of activity on your network is crucial in identifying the real threats to your network. The more log sources data collected– the better. However, it may not be necessary for your organization to monitor logs from all systems. Which devices you collect and monitor logs from should depend on your organization’s tolerance for risk.  An organization with a very low risk tolerance may decide to monitor logs from every device on their network. While another may have a very high tolerance and monitor logs from a only a handful. Most organizations fall somewhere in between, monitoring logs from their security devices and critical servers.

The next step required for Log Monitoring is aggregation. Logs contain a lot of data that is not needed for security analysis. As a result, they need to be filtered, normalized and condensed to minimize the “noise” and focus on security activity. This must be done for log monitoring to be performed in real-time.

Once logs have been aggregated, they can be correlated and assessed to identify security incidents. Correlation should incorporate all relevant log data, such as source IP, destination IP, summary, etc., and it should be performed across all log sources. Logic engines that leverage advanced correlation rules and statistical analysis streamline the correlation process; however, human analysis and assessment is equally important. A trained analyst’s capacity for inductive reasoning, combined with deep security experience and expertise, is vital to assessing genuine security incidents.

This is the point in the log monitoring process in which people become more important than technology. Events have been normalized, filtered, and correlated, and those remaining have a much higher probability of being a genuine security threat. Now, human intelligence must be applied to handle each of the events and judge if response is necessary. Human analysis should follow a mature Incident Handling Process, based on best practices and guidance from organizations such as the SANS Institute. This process should address high level analysis and categorization of threats for standardized assessment, but it should also give analysts the flexibility to fully investigate threats based on their training, intelligence, experience, and instincts.

Once a threat has been assessed, measures should be taken (if needed) to mitigate the risk the threat poses. Depending on the severity of the threat, response can range from simply logging the incident to immediate intervention. Regardless of the response, it should be based on clearly defined processes and policies to ensure consistency. What your policy calls for should depend on its tolerance for risk as it pertains to the confidentiality, availability, and integrity of sensitive data and IT assets.

The most critical aspect of the log monitoring process is a feedback loop that enables continuous improvement. Ideally, all of the information learned while monitoring logs is fed back into the process in real-time to ensure optimum effectiveness and efficiency. For example, there should be mechanisms in place to allow the analysts to easily create new filters and correlation rules. If there is no ongoing effort to take what has been learned and apply it to the log monitoring process, you will have to add more and more people to the process as the proportion of events requiring human analysis grows unchecked.

Log Retention

As with Log Monitoring, the Log Retention process begins with collecting logs from the systems on your network. But once logs are collected, the process for retaining log data differs significantly from monitoring log data. Because the purpose of Log Retention is to store and archive all raw log data so that it is forensically sound, the logs cannot be altered in any way prior to being retained. This means that logs cannot be filtered to remove data that is not relevant to security and they cannot be normalized. This makes Log Retention a challenge for many organizations, as the sheer volume of raw log data can overwhelm their capacity to process and store the logs.

Logs must be parsed and indexed in order to search through the large volumes of raw log data. Ideally, logs should be parsed as they are inserted into the retention database. This will minimize the amount of reprocessing that has to be done when you query the logs later, allowing for high performance regardless of the amount of log data being searched.

Storing raw logs in their entirety is essential to retaining forensically-sound logs, but it does require having sufficient disk space to handle the large volumes of log data. Using industry-standard compression algorithms helps to minimize the storage space needed while maintaining the integrity of the raw logs. Many organizations also leverage their existing Storage Area Network (SAN) or Network-attached Storage (NAS) to cost effectively store and archive their logs. Checksums and strong encryption should be used to ensure the integrity of stored log data.

Reporting

Both the Log Monitoring and Log Retention processes provide access to information that essential to demonstrating the effectiveness of your security controls and proving compliance with policies and regulations. Log Management brings together security and audit data from across your organization, streamlining the reporting process and saving time for you and your auditors. Reporting also provides transparency into the Log Monitoring and Retention processes, allowing you to verify their effectiveness and make modifications where necessary.

Additional Resources:

 

Featured Report: Best Practices in Choosing and Consuming Managed Security Services

All Best-in-Class Companies use some managed security services and the number one reason organizations are pursing managed security services is to improve their security. Best-in-Class organizations report fewer security incidents, fewer malware infections, fewer incidents of data loss, greater reduction in fraud, fewer failed audits, and greater reduction in help desk costs associated with security events.

View this complimentary Aberdeen research report

Security 101: Web Hosted Malicious Code

Description

The target in this architecture is a client browsing the web.  A criminal configures a web server to host malicious code that is downloaded to the client and executed with or without user interaction.  The code may be an executable that the user would give permission to execute or javascript that includes an exploit that exercises a vulnerability in the browser or other browser helper applications such as flash or quicktime.  The server could be a machine that is owned by the criminal or more likely a machine that the criminal has compromised.  If it is a machine that has been compromised the hacker will leave the look and feel the same and add an HTML iFrame that redirects the browser to a different server hosting the malicious code. This architecture makes popular web servers risky because of the potential number of desktops that could be compromised if a popular web server hosts malicious code.  Social networking websites such as MySpace and Facebook as well as sites hosting blogs are useful to the attacker because the content hosted on the website is provided by the users.  In this case the criminal can register with the website and submit malicious content without having to compromise the website.  

Objectives

The objective of the attacker is remote code execution on the client.

Trust Model

The client trusts that the server does not host malicious code and intends no harm.

Strengths

Ability to detect: 5 (High)  Ease: 1 (Easy) It is fairly easy for a criminal to configure a webserver that hosts malicious code. It becomes a little more difficult if the criminal attempts to compromise a web server.   

Weaknesses

In order to compromise a client the user must first browse to the website. 

Detection Points

This attack can be detected at the server to see if there is malicious code and/or that web pages have been changed. This attack can be detected at the client before the code is executed or somewhere in the network.

Prevention Points

In order to protect the client use a reverse proxy with URL content filtering to filter out known bad sites, network intrusion prevention for filtering malicious javascript and web browser and helper application exploits.

Evasion Techniques

There are several publicly available javascript obfuscators and packers that are used to evade detection and prevention.

References


SecureFacts

Regulatory requirements for logging

regulation table

 

All third-party brands and trademarks referenced in the text above belong to their respective owners. SecureWorks' On the Radar Newsletter is not authorized by, associated with or sponsored by the respective trademark owners.

Join Newsletter