PART 1 - Why Cybersecurity Needs Better Measurements for Risk
They start out the chapter telling us what the most important measurement is -- "We propose that the single most important measurement in cybersecurity risk assessment, or any other risk assessment, is to measure how well the risk assessment methods themselves work."
Risk analysis "works" when it measurably reduces risk.
In a 2005 study by Philip Tetlock, conducted over a period of 20 years, he concludes, regarding algorithms and experts: "It is impossible to find any domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical ones."
That is to say experts are not as good as algorithims at risk assessment.
It is difficult, even for "experts," to make accurate predictions. An example cited in the book is a 2010 study that asked CFOs to provide estimates of the annual returns on the S&P 500."These estimates were in the form of ranges--given as a lower bound and an upper bound--of values that were wide enough that the CFO believed they had an 80% chance that the range would contain the correct answer." However, they were found to be correctly in the range only 33% of the time.
The authors state that estimating subjective probabilities can be improved, a topic later in the book.
The last thing I'll pull out from this chapter are some of the conclusions they come to from the research: