Intelligence Analysis: Art or Science?
Check out this paper titled “Intelligence Analysis in Theater Joint Intelligence Centers: An Experiment in Applying Structured Methods” (I know, sounds like a real page turner, but keep reading, it’s worth it).
There’s a debate within the intelligence community (well, ok, debate is probably a strong word – discussion? lone ramblings of a deranged mind?) about whether analysis is more like an art or a science. Depending on where you fall within that debate determines quite a bit about how you go about conducting analysis and how you view the ability of others to become analysts.
Of those who believe it’s an art (and I was firmly in this camp for a number of years), intelligence analysis is seen as the result, primarily, of imagination and inspiration. While subject matter expertise is important, good analysts were seen to possess some, intangible ‘X-factor’ which allowed them to look at data in ways mere mortals weren’t able to and ‘voila’ out would pop some hidden wisdom. I think there’s a number of reasons why that view has a lot of appeal for analysts and non-analysts alike. For analysts, it feeds their egos to be in a select group that can peer behind the curtain and see things that most cannot. Besides, books, movies and TV have, for decades if not centuries (think Sherlock Holmes) have fed into this view.
While that paradigm may have worked for quite a long time a host of factors over the past twenty years (the internet, the rise of the 24 hour news cycle, the rise of the ‘punditocracy‘ and, of course, my old nemesis Tom Clancy) have allowed virtually everyone to have access to the same information and led people to believe that with access to information, anyone can conduct analysis off the cuff.
This has proven a serious obstacle in both law enforcement and military circles. As Folker writes (and no, it’s Folker, not Focker)
Decision-makers typically form conclusions based on intuition developed through years of experience. Therefore, the intelligence analyst who provides an assessment based on inexperience and his own (lesser developed) intuition should not expect to be taken seriously.
In my own experience in Afghanistan, I found that decision-makers didn’t even need years of experience to form conclusions. In many cases an affinity for the TV show ‘24‘ and too much time watching Bill O’Reilly was sufficient to come up with analysis that would trump (at least in the mind of the decision-maker) anything an analyst could produce.
So, this problem of perception (can analysts really add any value given their lack of ‘real world’ experience when compared with soldiers/cops/decision-makers who’ve been on the front-line/streets/boardroom for years?) is a serious, if relatively recent, one. But, our troubles don’t end there…
The next question is: “Regardless of how analysts are perceived, does the ‘analysis as art’ methodology (such as it is) produce good results? Are there other methods that produce better results?” If the answers to those questions are ‘yes’ and ‘no’ respectively, than we only have a problem of perception (essentially a community wide Cassandra Complex). Not easy to overcome, but at least there would be some satisfaction in knowing we were getting the right answers.
Unfortunately, we don’t really know the answers to those questions and that’s what Mr. Folker addresses. He argues that a structured approach to intelligence analysis leads to better answers. It’s been awhile since I’ve taken statistics and his sample size is small (unavoidably so, I think) but his results are quite interesting.
It seems to me that as long as structured analytic methods can’t be proven to produce worse results than the intuitive, ‘analysis as art’ methodology they should be incorporated into standard procedure for the law enforcement and military communities. Even if the two methods produce the same results, a structured methodology will give the analysts additional credibility by being able to demonstrate how conclusions were reached and what evidence was used. When dealing with a skeptical audience, hard evidence always beats a wave of the hands and theatrical ‘Voila!’.
I’ve moved away from the ‘analysis as art’ camp over the past couple of years into a middle ground (I might even say logom) that places analysis into a category that must draw upon both intuitive and structured systems in order to produce quality products. We’ll never get to a point where we can make a checklist that people can follow to produce analytical products (and I’m not sure we’d want to even if we could) but, by the same token, we need to get beyond just ‘wingin’ it’ as well.