In a previous post I started a discussion of stating intelligence estimates in terms of percentage confidence. This was based on Michael Schrage’s op ed piece: What Percent Is 'Slam Dunk'?
Give Us Odds on Those Estimates. Shrage wrote that it wasn’t enough for George Tenet to assure the President that the case for WMD’s in Iraq was a “slam dunk.” Estimates should be quantified. The intelligence community has been trying this for years, as Schrage writes.
Part of this effort included training in subjective probability statements. During the mid-70’s I took a subjective probability test. The test consisted of 200 factual questions. For example, “in 1934 was the per capita murder rate higher in Los Angeles or Chicago?” We were to answer the questions and then indicate our confidence in the correctness of the answer. If we didn’t know, we would indicate 50 percent confidence. If certain, we would indicate 100 percent. It turns out that many analysts are good at estimating 50, 60, 90 and 100 percent confidence levels. It is possible to know when we are uncertain (50%), slightly certain (60%), almost certain (90%) or cerain of our knowledge of facts.
Factual questions, however, do not have the same emotional content as questions that arise in intelligence estimates. An analyst might be very good at stating subjective probability on this test and be terrible on a highly controversial question such as the existence of WMD in Iraq.
Sometimes decisions are made quickly, without any quantitative analysis – or in contradiction to it. This phenomenon is explored in Malcom Gladwell’s new book Blink : The Power of Thinking Without Thinking . I contend that one’s ability to make good snap decisions is related to the the psychology of mind-body integration.
History gives us cases of leaders rejecting quantitative estimates. Robert E. Lee refused to consider a high confidence estimate that he would be unable to defend Richmond against the Union Army. Jefferson Davis relates the story:
In the early days of June, 1862, General McClellan threatened the capital, Richmond, with an army numerically much superior to that to the command of which Lee had been assigned. A day or two after he had joined the army I was riding to the front, and saw a number of horses hitched in front of a house, and among them recognized General Lee's. Upon dismounting and going in, I found some general officers engaged in consultation with him as to how McClellan's advance could be checked, and one of them commenced to explain the disparity of force and with pencil and paper to show how the enemy could throw out his boyaus and by successive parallels make his approach irresistible. “Stop, stop,” said Lee, “if you go to ciphering we are whipped beforehand.” He ordered the construction of earthworks, put guns in a position for a defensive line on the south side of the Chickahominy, and then commenced the strategic movement which was the inception of the seven days' battles, ending in uncovering the capital and driving the enemy to the cover of his gunboats in the James river.
In the end, I think Schrage is correct. As a matter of the ethics of intelligence, analysts ought to tell decisionmakers exactly how certain they are. They also need to forthrightly state that some questions are not answerable by quantitative methods. In doing so, they will at least identify areas in which "thinking without thinking" wil be rquired.
Comments