A Look at How to Measure Anything in Cybersecurity Risk (Chapter 7)

by Phil Conrad

PART 2 - Evolving the Model of Cybersecurity Risk

Chapter 7 - Calibrated Estimates: How Much Do You Know Now?

"A range that has a particular chance of containing the correct answer is called in statistics a confidence interval (CI). A 90% CI Is a range that has a 90% chance of containing the correct answer."

The 90% CI is widely used in the methodology the authors demonstrate to assign probability. This chapter teaches us that we can get better at quantifying our uncertainty. The authors do this through the use of calibration test samples. Through the use of even these small samples, we (the reader) are able to get an idea as to our subjective confidence level. The two extremes of subjective confidence are overconfidence - overstating knowledge and being correct less often than expected, and underconfidence - understating knowledge and being correct much more often than expected. They go on to describe how to determine your overconfidence or underconfidence on particular answers or as a whole.

What the research finds in these calibration tests is "most people are providing ranges that are more like 40% or 60% CI, not a 90% CI." However, people can be calibrated through training. "Most people get nearly perfectly calibrated after just a half-day of training."

The methods they share to improve your probability calibration include:

• Repetition and feedback - take several tests in succession, assessing how well you did after each one and attempting to improve your performance on the next one.
• Equivalent bets - for each estimate, set up the equivalent bet to test if that range or probability really reflects your uncertainty.
• Consider two pros and two cons - think of at least two reasons why you should be confident in your assessment and two reasons you could be wrong.
• Avoid anchoring - think of range questions as two separate binary questions of the form "Are you 95% certain that the true value is over/under (pick one) the lower/upper (pick one) bound?
• Reverse the anchoring effect - start with extremely wide ranges and narrow them with the "absurdity test" as you eliminate highly unlikely values.

Through various training they've done for research and for industry, the difference in accuracy is due entirely to calibration training.

This chapter was quite convincing that a person can indeed be trained to provide more accurate estimates, particularly if that is a responsibility in their job.

Home