Logging of data from IT devices and systems of all types generates huge volumes of data that is often never reviewed or used effectively. Yet there is tremendous value in those logs.
"Logs are underrated… Never underestimate the ability to combine log messages from multiple sources to paint a single, more informative picture."
That's just one of the many pieces of solid advice and insight from a new book entitled "Logging & Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management," by Kevin J. Schmidt, Christopher Phillips and Dr. Anton Chuvakin. Schmidt and Phillips are Dell SecureWorks staff members. Chavakin is a research leader with Gartner, Inc., and a former exec with LogLogic and Qualys. The book was just released this month.
I interviewed Schmidt, a senior manager on our Engineering Team, about the book, which has been nearly seven years in the making. Schmidt recognized that there was a need for a book like this to help inform IT managers, system administrators, network administrators, compliance managers and a range of others who are not currently experts on logging but who need to understand logging better. The book is not written for logging experts, Schmidt said, but I believe even many experts would find it useful.
"Logs contain a wealth of hidden riches," is another nugget from the book.
I think of logging as being one of the basic blocking and tackling skills that every organization concerned with security and compliance needs to have. The "hidden riches" are insight and context and understanding. In many cases, the proper utilization of log data can let you discover malicious activity, compromises, threat attack methods used, whether the attack succeeded or failed, the timing of events, etc. Interestingly, many of the insights available from far more expensive security technologies are available in log data --- if --- and it's a big if --- you can use the log data effectively and correlate different pieces of data to provide insight.
If you believe your logging process is not doing what it needs to do, consider the kind of log monitoring and/or log retention services Dell SecureWorks provides, which put that log data through a powerful correlation engine to be sure that customers do get insight and alerts from the data.
It's surprising that no similar books have been written on this topic in the last 8 to 10 years. What's changed in that time frame?
At least two things:
- The volume of log data has exploded and continues to grow dramatically
- The adoption of cloud technology has done the same thing
The good news is that new tools using cloud technology, such as Sumo Logic and Loggly are helping make it easier to handle log data in huge volumes and to get insight from the data.
What does Schmidt hope the main take-away is from the book? "That your logs can help you understand what's going on and help you with compliance issues as well, and that the book can help make it easier for you to collect and analyze your logs."
What are your thoughts about logging, the value of log data and what your main challenges are in logging and log management?