When Peer Review Becomes Peer Redo, Pass Through
We are all likely to experience it at some point in our intelligence careers. The product or presentation you have been working on for weeks or months is ready for the review process. Maybe your supervisor calls you in and wants to review it with you directly. Or perhaps your office has a standardized system where you simply pass products around so that a minimum number of eyes have at least given it a once-over. Regardless, you are anxious to publish and only a couple of peer reviews stand in your way.
Then you get the document back and realize that something in the peer review process must have went horribly wrong. You slump in your chair and start taking stock of the reviews that have just come in to your desk. One of two things is likely to have happened, and depending on your personality you probably loathe one more than the other.
The first scenario involves the proverbial red pen. Your product has been slashed and burned. There are squiggly lines all over it pointing from one paragraph to another with illegible notes in between drawings and annotations running off the sides of the pages. There are more words crossed out and rewritten than remain untouched.
There appears to be some kibitzing between peers within your document, with reviewers commenting on comments left by other reviewers similar to the comments section of an online news article. There is also a small chance that somewhere in the process your product was used as scratch paper by a reviewer to make a note about a contact or meeting.
You are left reeling and wondering what to do next. “I asked for a peer review, not a peer redo,” you might think to yourself. Your product is now a circular mess with little direction that leaves you wondering if you should just start over. Demoralized and disenchanted, you take a break and then begin processing the comments, scheduling follow-up meetings to decipher the handwriting, and promising yourself you will never write a product like this again.
The second scenario can be just as defeating. You have collaborated on a big project and now you need trusted colleagues to give you honest feedback. The product is mostly there, but something is missing where a fresh perspective would help identify any weaknesses or areas for improvement. You get the document back and you flip through the pages searching for inspiration and insights. Then you flip through them again, and again. Did they use invisible ink?
You realize that for all your weeks of hard work you received zero feedback. Not even an errant pen mark. This could be for many reasons. Maybe the product is that good. Just as likely, however, your peer reviewers read it like they would a congressional resolution commemorating Earth Day – hardly at all. What is worse is that you spot at least one misspelling or irregular font that made it through your own review that should have been easily recognized by any peer reviewer. “I asked for a peer review, not a peer pass through,” you might think to yourself.
So how can you improve your chances of avoiding these two peer review traps? Be mindful that some of the issues uncovered during a peer review should have been addressed long before the review stage of the production process.
For example, if a customer (likely leadership) has tasked you with an intelligence need or assignment and you find yourself saying “I hope this is what he wants” you are increasing your chances of having your white draft returned to you covered in red. Prior preparation prevents poor performance, and you need to be sure you understand the customer’s needs and your audience. Guessing is not going to somehow channel what the customer actually wants. If you choose to go ahead and complete a product not fully understanding the customer’s request, be prepared for the possibility that you may be writing it twice.
It sounds counterintuitive, but analysts need to be sure they know that the customer knows what s/he wants. It is vital that you keep an open line of communication with your customer so that you can realign expectations whenever you (or s/he) feel that the analysis and production is not meeting the intent of the original request. If after some time of analyzing and producing the customer’s request you still feel unsure about what exactly s/he wants, schedule another meeting to present what you have so far to make sure you are on the right track. If you do not have regular access to your customer, submit a request through your chain of command. It will save both you and him/her a lot of time and frustration before the product is written.
In some ways, scenario two can be more difficult to resolve. Your approach may be entirely up to your preferred interpersonal conflict management style. I have some general suggestions, however.
One of the great attributes about the intelligence profession is that it is stimulating on many levels. Intelligence analysis in particular involves a high degree of creativity, innovation, collaboration, critical thinking, analysis, and prediction. All of these are nurtured through two-way communication among intelligence professionals: supervisor-to-employee, mentor-to-learner, peer-to-peer, customer-to-producer. When those relationships break down there is a disruption in the communication that is crucial to successful intelligence and analysis.
If you find yourself in scenario two and you think it is not the result of perfection, say something to your supervisor or colleague. Explain that you recognize the importance of the peer review process in intelligence production. Make sure your peers know you were disappointed in the effort you felt they gave in their reviews, and that you value their expertise and knowledge. Offer suggestions on how you can facilitate successful peer review with them, such as scheduling a time to look over your product together or accepting feedback through a different medium. Finally, suggest to your supervisor or leadership that an office-wide refresher for peer review standards and training may be warranted. Peer review is a learned and practiced skill and can be just as important as analytic techniques and tradecraft. It should not be optional or completed haphazardly.
It is also beneficial to clearly delineate roles and responsibilities for peer reviewers before they begin tearing through your document. Do not be shy about outlining up front what you are looking for from each reviewer. If you just want a legal review then say that. Feel comfortable with the spelling and grammar but need someone to challenge your key findings and analysis? Make sure your reviewer knows that. You may increase your feedback and receive more tailored commentary if you give your reviewers a narrower scope, especially with long documents. This approach also personalizes the review, and helps prevent your peers from thinking that others will provide better feedback than them so why bother or that there is too much to review altogether.
While most of your colleagues will appreciate and be trained in the craft of peer review, you may be unable to escape these two scenarios completely. Strong peer review is critical to upholding quality production, analytic standards, and sound intelligence practices. I hope this has given you some ideas for the next time you receive a dreaded peer redo or peer pass through.
Posted on 6 August 2011, in Decision Making, Intelligence Analysis Techniques and tagged analysis, critical thinking, intelcloud, intelligence analysis, peer review, terrorism. Bookmark the permalink. 1 Comment.