For the past decade, I have had this notion that there must be a Cybersecurity Canon: a list of must-read books where the content is timeless, genuinely represents an aspect of the community that is true and precise and that, if not read, leaves a hole in cybersecurity professional’s education. I’ll be presenting on this topic at RSA 2014, and between now and then, I’d like to discuss a few of my early candidates for inclusion. I love a good argument, so feel free to let me know what you think.
Security Metrics: Replacing Fear, Uncertainty and Doubt (2007) by Andrew Jaquith
I have been interested in cybersecurity metrics and how to visualize them since before we were connecting the Internet with strings and soup cans. In 2011, I had been looking for somebody to put some rigor to the idea when I stumbled upon a strong, positive review of Andrew Jaquith’s book on Amazon. A little more digging told me this was a book I really should check out.
From the beginning, Jaquith attacks the security community’s sacred cow of applying annualized loss expectancy (ALE) to convince management that the security program it is paying for is working. I have to say that I loved this attack. I remember first learning about ALE when I was studying for the Certified Information Systems Security Professional (CISSP) exam back in the day. I thought then that ALE sounded well and good when you said it fast, but in reality, you were just making up the numbers to plug into a formula that sounded scientific.
According to Jaquith, and most every CISSP preparatory exam book on the planet, “ALE is the monetary loss that can be expected for an asset due to a risk over a 1-year period and is calculated by multiplying the single loss expectancy (SLE) by the annualized rate of occurrence (ARO).”
Doesn’t that sound precise and mathematical? Indeed it does. But it turns out that there are lots of problems with this formula. The biggest problem is that we don’t know what the probabilities are. How can we possibly know what the probability is that an advanced-persistent-threat-style attack will compromise the computer that your chief of counsel’s secretary uses? This is not the insurance industry; we do not have actuary tables derived from decades of data collection that can tell us precisely what these adversaries will do, how often they will do it and how much it will cost us when they do it.
So what, Jaquith and others have asked, do ALE practitioners do in the absence of hard data? They guess. They estimate. They fudge. And when they do this, they undermine the veracity of the very process that they are trying to convince management is so exacting. What good is a scientific formula if all you do is fill it with garbage data?
Jaquith’s thesis is that, instead of using imprecise models like ALE, security professionals should use metrics instead. He says that “[this change in thinking] requires practitioners to think about security in the same way that other disciplines do – as activities that can be named, and whose efficiencies can be measured with key indicators.”
Coincidentally, the first time I read Jaquith’s book, I just happened to listen to the Patrick Gray Risky Business podcast from April 2011 where he interviewed Brian Snow. Snow is a former NSA information assurance technical director, and he had a lot to say then about the folly of using probabilistic risk assessments, like ALE, to improve the cost-effectiveness of securing nuclear facilitates and government information assurance programs.
Snow made the point that these models are fine for standard risks that routinely occur—like what is the mean time to failure of the hard drive in your laptop—but that they fail miserably when trying to predict cases that have high impact to an organization but are not likely to occur. These cases that Snow referred to are called “black swan events.”
Black Swan Events
The “black swan event” term was made famous by Nassim Nicholas Taleb in his 2007 book “The Black Swan: The Impact of the Highly Improbable.” For some organizations, computer breaches are black swan events that Taleb describes as “outliers that carry extreme impact.” They are outliers because the chances of something like that happening to your network are pretty small, but when it does, the cost to your organization is extreme.
Jaquith’s solution is to “… quantify, classify, and measure information security operations in a modern enterprise environment” and to provide “… a set of key indicators that tell customers how healthy their security operations are.”
He spends a good portion of his book, two entire chapters actually, explaining what some of these metrics might be. Your organization might not have a use for all of them, but you will appreciate the thoroughness that Jaquith uses to explain why they should be considered.
As a bonus, he spends a chapter reviewing the fundamentals of statistics. If you are like me and slept through your probability and statistics course in college, you will welcome this refresher. Jaquith’s simple explanation alone about what a standard deviation is and what correlation really means is worth the price of admission.
As an extra bonus, he spends a chapter on visualization. I am a fan of Dr. Edward Tufte, who is in my opinion the world’s leading expert on how to visually display complex data. Tufte devotees will learn nothing new here but will appreciate how Jaquith reduces Tufte’s four seminal books on the subject to six rules:
- It’s about the data, not the design
- Just say no to three-dimensional graphics and cutesy chart junk
- Don’t go off to meet the (Microsoft) wizard
- Erase, erase, erase.
- Reconsider Technicolor
- Label honestly and without contortions
The only real fault I have with the book is the last chapter, “Designing Security Scorecards.” Here, Jaquith had the opportunity to show some practical security dashboards that perhaps some real organization used and found useful. Instead, he spends the entire chapter explaining what goes into making a scorecard.
As I got closer to the end of the book, I just knew that I was going to see some dazzling examples that I might use in my own organization. When I turned to the last page and found nothing but the index, I was dumbfounded. He provided no examples of real-world security dashboards. D’oh! So close to being perfect!
Why It’s Worth It
That one caveat aside, Jaquith’s book is well worth the read. I recommend it highly. I dare you to get to the end of that book without learning something that will help you in your current job, and even if security metrics are not your thing, then statistics and visualization will make you a more well-rounded business person.
But for you security professionals out there, this book is for you. It will help you unshackle yourself from the chains of probabilistic risk assessments. It will turn you away from the dark side and toward a more meaningful process to assess your enterprise’s security. You should have read this by now.