CAMBPELL SYSTEMATIC REVIEW PROTOCOL COVER SHEET TITLE: Strategies Related to the Prevention, Detection, Management and Response

to Terrorism: A Campbell Systematic Review.

AUTHORS: Dr. Cynthia Lum, Assistant Professor Northeastern University College of Criminal Justice 415 Churchill Hall Boston, MA 02115 (617) 373-4076 c.lum@neu.edu Dr. Leslie W. Kennedy, Dean Rutgers University School of Criminal Justice 123 Washington Street, Room 568 Newark, NJ 07102 (973) 353-3311 kennedy@andromeda.rutgers.edu. Ms. Alison Sherley Rutgers University School of Criminal Justice 123 Washington Street, Room 568 Newark, NJ 07102 (973) 353-3451 asherley@pegasus.rutgers.edu

SOURCES OF SUPPORT: Cynthia Lum is supported by Northeastern University as well as previously by the Rutgers Center for the Study of Public Security at Rutgers University-Newark. Leslie Kennedy and Alison Sherley are supported by Rutgers Center for the Study of Public Security at Rutgers University-Newark.

BACKGROUND FOR THE REVIEW

INTRODUCTION Social scientists have often responded to events, crises and public concerns through research. In crime and justice, for example, the increase in crime rates and imprisonment in the 1970s and 80s (see Blumstein and Wallman, 2000) and the debate over the effectiveness of rehabilitation (Lipton, Martinson and Wilks, 1975; MacKenzie, 1997) led to a large amount of research on imprisonment, sentencing schemes and alternative corrections (Blumstein, 1988). Similarly, heightened awareness of domestic violence and outrage towards police responses to these incidents resulted in increased attention to the issue by criminologists and later to policy responses reflecting research (Sherman, 1992). This same policy focus and attempts to influence current policies can be seen in research on high-profile concerns about gangs and drugs (Lane and Meeker, 2000; Laniel, 1999; Reuter, 2001) or victimization and fear of crime (Farrall, Bannister, Ditton and Gilchrist, 1997; Sebba, 2001). Many of these responses to crises in criminal justice have centered around two related concerns: implementation and development of effective policy to counteract the crises, or the evaluation of the effectiveness of current policies. Both have become generally encompassed under evidence-based crime prevention approaches (see MacKenzie, 2000; Sherman et al., 1997; Sherman, Farrington, Welsh and MacKenzie, 2002) . Evidence-based crime prevention has been facilitated by a vast amount of research on what works in crime prevention (see Lum and Yang, 2004; Sherman et al., 2002), meta-analyses and systematic reviews that attempt to organize , make sense of, and critique research, and the establishment of the Campbell Collaboration (and specifically its Crime and Justice Coordinating Group) which encourages

these endeavors. This movement has served important functions in improving both the quality of research as well as properly informed practice in areas such as police patrol, corrections-based schemes, juvenile diversion, treatment of sex offenders, and sentencing practices. Evidencebased perspectives can also provide a moderating effect on policy-responses to crises that have become more influenced by moral panics (Cohen, 1972) than by reason or facts. For example, the use of boot camps to control the juvenile superpredator has been widely discounted by much research evaluating the effective ness of these programs (see Gover, MacKenzie and Styve, 2000) counter-balancing the popular movement towards utilizing these approaches in controlling juvenile crime. The current public concern over terrorism also presents similar controversies and research challenges. While counter-terrorism and terrorism prevention programs have blossomed considerably since September 11th , there has been little evaluation of either the effectiveness of the wide array of anti-terrorism policies or the effects of the responses themselves. Given the vast array of different strategies designed to fight terrorism, we know very little about which of these might actually be effective and appropriate. Millions of dollars are being expended on potentially fruitless programs, or worse, harmful responses that may promote or benefit terrorists or harm populations generally. As with boot camps or police strategies to deal with domestic violence, evaluating the effectiveness and effects of strategies geared toward countering terrorism is part of ensuring that policy is informed, evidence-based and effective. Yet, evaluating the effectiveness of anti-terrorism strategies remains a challenging and controversial task for a variety of reasons. Unlike crime, terrorism incidents have not been easily defined and therefore, strategies may be difficult to pinpoint. If we include all political violence

as terrorism (including ethnic conflict) then we might also have to include strategies such as war, government suppression, laws that manipulate boundaries or peace treaties as possible strategies to evaluate. Even more problematic may be political goals and biases that underlie terrorism research. For example, one might argue that anti-terrorism effectiveness is measured when there is political suppression of opposition parties, which may not be valued in democratic societies. Other strategies may seem effective but may be in gross violation of human rights norms, such as the inhumane treatment of prisoners or other corrective policies that may not be acceptable in many societies. Additionally, like much of crime, political violence involves multiple points at which interventions may be effective and where outcomes might be measured. For example, antiterrorism strategies may include prevention and alleviation of early risk factors, situational prevention of actual events, or post-event responses. Furthermore, because of the rare nature of terrorism events, it may also be difficult to determine whether these strategies actually worked. Thus, researchers may be interested in detecting secondary effects of prevention strategies such as the reduction of the fear of terrorism or the adequate and efficient response after an event occurs. Other strategies may focus on detecting potential terrorism events or other high risk situations. The purpose of this review is to organize and comprehend the research on strategies designed to prevent, detect, manage and respond to terrorism and political violence in order to make sense of the focus and quality of this research and to make future recommendations. Despite the aforementioned challenges about research on anti-terrorism strategies, conducting a Campbell systematic review is a relevant pursuit for two reasons. The first is obvious given the explosion of anti-terrorism policies and the millions of dollars spent on them, we need to

know what works, doesnt work or is promising (see Sherman et al., 1997) in terms of the most optimal use of resources. This review will function as a mechanism to systematically collect and comprehend the scattered research on terrorism that takes place across multiple disciplines and countries. Secondly, a systematic review may also expose what little we really know about the effects and effectiveness of anti-terrorism strategies and areas of possible future research. This may not only shape research agendas but may also shape more thoughtful policy development as well. Currently, the state of terrorism research, although vast in many respects, does little to critically evaluate the effectiveness and effects of programs designed to combat terrorism and we have already seen harmful effects of policies which we know little about.

A PRELIMINARY REVIEW OF TERRORISM LITERATURE Before approaching a review of anti-terrorism strategies, a preliminary and general examination of terrorism research can provide a better understanding of the scope of the research. From time to time, a few attempts at reviews of terrorism research have been undertaken (see, e.g. Halkides, 1995; Hoffman, 1992; Miller, 1988; Romano, 1984; Schmid and Jongman, 1988; Silke, 2004), although September 11th has certainly necessitated the need for an updated review on strategies, specifically. To conduct this preliminary review , we focused on published articles from multiple disciplines. 1 To include as much research as possible, seventeen separate literary databases were searched in January 2003 for broad and sweeping terms related to terrorism and political violence. 2 The universe of literature was multidisciplinary, including subjects related to economics, gender issues, criminal justice, education, geography, the humanities, medicine, psychology, political science, public policy, sociology and the law. We located 14,006 articles from these databases after excluding for duplicate records.3 For each

article located, the author, year, title, citation, abstract and keywords were recorded. Because of the volume of the initial literature search, publications were first divided according to the review status of their sources and only abstracts from peer-reviewed sources were individually read. Again, it should be emphasized that the filters used for this preliminary review will be re-evaluated when searching only for research on the evaluation of anti-terrorism strategies. The focus on peer-reviewed sources was not only practically necessary due to the volume of the literature, but served as a preliminary filter for understanding the quality and substance of academic research on terrorism and political violence. Many of the articles from non peer-reviewed sources were either news reports, opinion-editorials, advertisements, announcements or bulletins that would not normally be considered research. To separate articles from peer-reviewed sources, a list of peer-reviewed journals was generated using peerreview lists compiled by EBSCO,4 OCLC,5 the American Medical Association, 6 and the American Association for the Advancement of Science,7 and by investigating individual journal titles if not listed in these databases. Each of the 14,006 articles was then matched with a peerreview status using a relational database program. 6,041 articles could be identified as coming from peer-reviewed sources. Clearly, a large amount of writing regarding terrorism appears in non-peer reviewed journals or in journals with unknown peer-reviewed status. The first, most unique finding compared to past literature reviews was that September 11th has had an unprecedented effect on terrorism research. As Figure 1 indicates, among the entire 14,006 journal articles, approximately 54% were published in 2001 and 2002. As many of the journals were not refereed and represented magazines and other opinion outlets, this trend could simply represent an emotion bias after the attacks. However, when only examining articles from peer-reviewed journals, the same trend was evident. Neither the Oklahoma City

bombing in 1995 nor any other significant terrorist event, including the World Trade Center bombing in 1993, has ever generated this much research interest. [FIGURE 1 HERE] This exponential growth of research reflects not only the response to September 11th by researchers, but also increases in federal funding for research and development related to terrorism over the fiscal years following the attacks (Guinnessy and Dawson, 2002; Issues in Science and Technology, 2002; Macilwain, 2002; Silke, 2004, generally). Although time will tell whether this substantial increase is a fad, this trend is indeed cur rently significant, pointing to the relevance of terrorism research generally. Furthermore, this increase in research has also been matched by the massive build-up of anti-terrorism efforts and bureaucracy, much of which we have little or no understanding as to their effectiveness or consequences. Thus, not only is interest and concern of terrorism real, but substantive and institutionalized changes have made the effect of September 11th significant. A number of questions come to mind with the exponent ial growth of this research in relation to the goals of a Campbell systematic review. How much of this research is actually evaluation research? What types of topics are being evaluated? What is the methodological quality of this research? What conclusions are being made from this research? Does this research reflect the types of policies that are being implemented? To begin to answer these questions, we analyzed only peer-reviewed articles to better estimate the overall state of evaluation research on anti- terrorism strategies. To ascertain the nature of the methods used to analyze terrorism, abstracts of articles from peer-reviewed sources were read and coded for the type of methodology used. While we recognize that the abstracts might not provide detailed information about the methodology used

for each study, the volume of studies did not allow us to individually read every article for this preliminary review. However, we anticipate that this limitation is a minor source of error that does not largely impact our general findings (and for the Campbell review articles will be scrutinized in-full). Three basic categorizations were preliminarily developed to facilitate further identification of articles that may be useful to a Campbell Review: thought pieces, case studies, and empirically-based research. Briefly, thought pieces were classified as publications that were philosophical, theoretical or opinion-based without any indication that a case study or empirical analysis of a topic was undertaken. For example, articles discussing problems with defining terrorism, legal issues surrounding 9-11, historical accounts of a particular conflict, descriptions of a particular emergency response or opinions about the governments policy towards counter-terrorism would be categorized as thought pieces. Case studies became a somewhat confusing category (as some case studies could be thought pieces and others empirical analyses), so only those publications in which authors specifically defined their work as a case study were included in this category. These publications tended to push the analysis of terrorism further than thought pieces by including in-depth description and qualitative analysis on a particular incident of terrorism or a particular situation or region. Publications described as empirical analyses were distinguished from thought pieces in that some form of systematic qualitative or quantitative analysis was undertaken on the data gathered. As it was expected that this would be the smallest of categories, a liberal definition of empirical was adopted and could include a wide range of methodological strengths, ranging from descriptive trend statistics to randomized controlled experiments. For example, articles which fell under t his category included the analysis of terrorist incidents across time, the effects of a particular intervention on the occurrence of terrorism, surveys on the psychological effects

of terrorism, attitudes towards the U.S. after September 11th, and multivariate analysis on factors contributing to the development of terrorist groups. Figure 2 shows the distribution of the type of methodology for all articles in peer reviewed publications where an abstract was available. Although thought pieces certainly have their own value in furthering the understanding of any topic, the research on terrorism has been bereft of empirical research, a problem which has been endemic in terrorism research since Schmid and Jongmans review in 1988. This problem has not improved after September 11th ; when only examining the research conducted in 2001 and 2002, the proportion of empirical or case studies was similar. [FIGURE 2 HERE] The dearth of empirical analysis in peer-reviewed sources suggests a number of possible concerns. First, practical problems in either qualitatively or quantitatively studying terrorism are often thought to be significant obstacles to overcome. Not only are events thought of as rare but units of analysis (individuals or groups who employ terror violence or actual events/incidents of terrorism) may be difficult to locate and study. Especially since much political violence occurs outside of our nations borders, a call for terrorism research presents similar difficulties that comparative criminological research does generally, including small sample sizes, difficulties in interpretation, and differences in measurements of key variables. More importantly, the scarcity of empirical analysis may also reflect the state of the science of terrorism research, which suggests that very little evaluation research on strategies designed to prevent, detect, manage or respond to terrorism events (or the potential of terrorism events) exists. We were also interested in a better understanding of what types of subjects were covered

in this research (and specifically how much of the general research reflected evaluation studies). To understand the distribution of subjects researched regarding terrorism, a coding schema was developed by initially taking a random sample of 200 articles from the peer-reviewed journals which were then individually examined to develop a preliminary list of topics. This was done so as not to force subjects into pre-defined categories. Then, each of the citations from peerreviewed sources that had abstracts were individually read and matched to this preliminary list, which was actively modified during this process. Thirty-five general categories initially emerged which were collapsed into seventeen categories shown in Table 1. Table 1 also reports the distribution of these categorizations for just those articles which were deemed empirical studies. [TABLE 1 HERE] A number of interesting findings emerged. Generally, issues related to weapons of mass destruction represented the largest proportion of articles (18.9%) followed by articles which focused on a specific issue, such as the Israel-Palestinian conflict, the IRA, Al Qaeda, or September 11th (if it could not be categorized elsewhere). Additionally, political responses to terrorism and the sociology of terrorism (causes, motivations, explanations, definitions) clearly dominate the research agenda of terrorism studies. These emphases most likely represent the political science bias of the study of terrorism and speak to some of the priorities that have been formulated. Although these matters are indeed salient, surprisingly absent from the literature were articles regarding subjects that are more likely to affect individuals, their fear, and the ramifications of anti-terrorism efforts. Given the rarity of terrorism events in American society, we are more likely to be affected by anti-terrorism responses than events themselves, concerns

that are also under-researched. This may include, for example, violations of civil rights, increases in law enforcement powers with questionable constitutionality, or post-traumatic psychological effects on victims. Little is known generally about the effectiveness of law enforcement and other non-political responses to terrorist events. Although legal issues have been more researched, they still represent only a small percentage of terrorism-related research. This is also true for issues related to coping with terrorism events and the general victimology of terrorism. Additionally, when thinking about future threats, such correlations between terrorism and religion, socio-economic factors and political responses have yet to be made. When examining those articles preliminarily deemed to be empirical, the findings are both encouraging and discouraging. While a quarter of the empirical work has been conducted on victimology, a subject relevant to our search for evaluation studies, and while it is anticipated that within the literature on weapons of mass destruction there may be empirical tests, we also found very little empirical work done on any type of response to terrorism or on the legal aspects of terrorism. The vast majority of empirically-based studies do not seem to evaluate the effects of anti-terrorism strategies. We expect that many evaluation studies will focus on the effects of treatment provided to victims of terrorism, the effects of anti-biological weapons programs and security related to airports.

OBJECTIVES OF A CAMPBELL REVIEW OF TERRORISM STRATEGIES These preliminary findings regarding terrorism research suggest a number of justifications for a Campbell review. Certainly, the study of terrorism is not simply a passing fad of little interest to criminologists. Not only has the research community begun to respond to the call for more information, but major institutional changes that have taken place since 9 have -11

10

provided important pressures to study a wide variety of terrorism-related issues. Government agencies, law enforcement and other first responders in major cities have changed their organizations to accommodate these concerns, for example, focusing on the possibility of responding to biological agents or to mass casualties. Both the social environment after September 11th as well as the Patriot Act 8 have affected and changed police perceptions of equity, due process, racial profiling and definitions of suspiciousness. Our court systems are now faced with individuals charged with a number of newly passed or recently revived antiterrorism laws. And, the mass mobilization of federal agencies pushes us to ask about their effectiveness, appropriateness and their relationship to our multi-cultural society. Many of these concerns have been matched by increased funding from granting agencies for terrorism research and anti-terrorism policy options (Guinnesy and Dawson, 2002; Macilwain, 2002) as well as calls to service by policy makers (Arlacchi, 2001; Parachini, 1999). Secondly, as a Campbell review will make more evident, the preliminary examination of the literature suggests that there is a dearth of empirical literature in terrorism research, suggesting that more evaluation research is needed. Much of this literature are thought pieces and do not address the effectiveness of strategies using more rigorous methods. One anticipated policy outcome of a Campbell systematic review will be to increase awareness regarding the lack of understanding of the multitude of strategies used to prevent the occurrence of terroris t acts, and to increase awareness of this for funders and policy-makers to encourage more rigorous evaluations. As with crime, the implications of strategies implemented to combat problems may actually cause more problems than anticipated. Effects could be harmful in that they might increase the possibility of terrorism or they might dilute the quality of other important values such as human rights, political or civil liberties. One very important need is the development of

11

informed, evidence-based terrorism policies, a mandate which has become an important part of research agendas (Sherman et al., 2002; Weisburd, Petrosino and Lum, 2003). This review suggests that social scientists can contribute through the evaluation of the effectiveness and effects of anti-terrorism policy towards more meaningful evidence-based approaches. In informing the public discourse about terrorism, particularly in the context of the major shift in discussions about security that impact on all parts of society, there needs to be a push for more quality research. Policy-makers need thoughtful, evidence-based, information utilizing available databases , as well as the continued search for and acquisition of further sources of data. The next sections outline the protocol for a Campbell systematic review of antiterrorism strategies.

METHODOLOGY Building upon the work already mentioned above, we propose the following methodology in accordance to the Campbell Systematic Review Protocol Guidelines.

CRITERIA FOR INCLUSI ON AND EXCLUSION OF STUDIES Terrorism research presents a number of difficulties for such a review. As already shown, what might be considered as an anti-terrorism strategy can include a wide variety of subjects, including political, social, legal, law enforcement, economic, preventative, reac tive or after-care responses. Limiting our search to only those responses which are law-related, for example, may exclude those responses that focus on risk factors. Certainly in other crime-related reviews of evaluations of anti-crime programs, such as the Maryland Report (see Sherman et al., 1997; Sherman et al., 2002), programs that may not be within a specified criminal justice

12

institution (e.g., the police, courts or corrections) are still crime-relevant (such as those strategies implemented in schools or by businesses or communities). Thus, we feel that a broad approach is warranted in creating the initial base of literature to examine. We propose to include evaluations of any strategy that involves the prevention, detection, management or response to terrorism events and incidents. While this may seem to be a difficult and disadvantaging task, our preliminary analysis shows that only 3% of our peer reviewed literature is empirically based, suggesting that the there may only be a couple of hundred empirically-based studies to begin with. While we will have to extend our search to 2003 and 2004 as well as look at all publication mediums (books, non-peer review materials, unpublished papers, government reports), we anticipate that the number of evaluations will be manageable despite our broad inclusion criteria. However, as already discussed, one main obstacle when seeking out evaluations of terrorism-related strategies is what we decide as the definition of terrorism. For our preliminary search, the goal is to be as inclusive as possible so as to not miss potential articles that may be used in the review. Thus, we will initially consider any evaluation research on terrorism which satisfies either of the following two criteria : 1) Any evaluat ion in which the outcome measured follows the official definition as set forth by the United States Department of State, which has described terrorism since 1983 (Title 22 of the United States Code, Section 2656f[d]) as premeditated, politically motivated violence perpetrated against noncombatant targets by sub-national groups or clandestine agents, usually intended to influence an audience; or 2) The author her or himself labels the outcome measured as terrorism, or related political violence eve n if not normally or traditionally considered as terrorism or even if it contradicts the definition of (1). While our intention of (1) is to provide some basic structure for the search, we do not wish to imply that only an American definition or perspective will be used (hence the 2nd possibility).

13

Following from these broad criteria, our search will encompass the aforementioned four categories of anti-terrorism strategies (although if another relevant category arises, we will be open to creating ne w categories): preventative, detection-oriented, managerial and responseoriented. We take this approach as we recognize that there are a number of points before, during and after a terrorist event in which strategies may be implemented (see Schmid, 1983; Walter, 1969). Prevention strategies are strategies designed to deter future events and might include airport security measures or target hardening evaluations. Studies on the preventative effects of legal measures (such as the Patriot Act) would also be included, for example. Research on detection strategies may include interventions related to, for example, airport security, but also might include evaluations of measures used to look into containers, border-related strategies, immigration policy or other tactics used to detect people, places, things or situations involved in terror-related activity. Managing strategies point to tactics after the fact and can include evaluations of the governments threat level mechanism, new hospital procedures designed to address issues of health security, or programs put in place to help people who have been impacted by a terrorist event, psychologically or otherwise (for example, evaluations of services intended to reduce general fear or victim post traumatic st ress). Evaluations looking at response strategies might focus on law enforcement responses to continual suicide bombing or the effects of economic sanctions on terrorism. It is anticipated that some of these categorizations will overlap. This Campbell review will also provide a unique opportunity for researchers to suggest a number of places where evaluation research may be needed. Because it is not expected that randomized or even quasi-experiments will be regularly present in the empirical research on terrorism, we intend to use a lower threshold than normally considered in a Campbell review to initially identify any possible study that might satisfy the

14

inclusion criteria above. Thus, the identification of any study based on empirical data will initially be considered. However, because this review will also have policy implications, we intend to specify our sample, within this group of empirical studies, to those studies that would satisfy moderate methodological requirements as described below. While we will use the Maryland Report Scientific Methods Score (see Sherman et al., 1997) of 3 as a general guideline, each study will be evaluated on its own. However, when reporting our results, we will remain extremely sensitive to the way we present our findings, emphasizing continuously that lower methodological rigor may lead to incorrect conclusions. Studies that will be excluded from this review will be thought pieces which have no empirical analysis of the effects or effectiveness of anti-terrorism strategies or that do not have any outcome measured for programs discussed.

SEARCH STRATEGY The search of these studies will build upon the comprehensive review as described above but will be broadened in a number of ways. All empirical-based research will be identified from the review above, which was based on a comprehensive search of multiple library datasets (as previously listed in Footnote 2). In addition to the data already collected, a second search of empirical literature specifically up to the current date will be undertaken, to assure that no study may have been missed since the last search which ended in 2002. Again, our initial search is intended to be as broad as possible as we do not anticipate finding a large amount of evaluation literature on anti-terrorism strategies. Furthermore, we have also compiled a comprehensive search of organizations and data-sources of terrorism research (see Appendices A and B) 9 which will be searched for evaluations conducted that may not be included in academic, peer reviewed

15

sources. Additionally, Alex Piquero, of the University of Florida, Gainesville and Gary LaFree, of the University of Maryland, College Park, have also conducted a review of hijacking which we will seek to add to our study with their permission. Other terrorism literature will also be searched, including evaluations from books and non-article mediums , government publications and unpublished material. We will also seek the assistance of a number of international scholars, who may be of assistance in locating research not present in these databases or that may not initially be in the English language. Finally, we will conduct an internet search to capture any other evaluations that may have been missed and that satisfy our me thodological criteria. In accordance with our broad definitions as mentioned above, we will conduct a widespread search of multiple terms related to terrorism and their derivatives (which may be added to or adjusted as required) . Some examples of these terms include : terrorism, political violence, political crime, emergency response, riots, anti-Semitic, white supremacist, black supremacist, militia groups, bio -terrorism, weapons of mass destruction, homeland security, national security, ethnic violence, bombing, suicide bombing, aircraft hijacking and skyjacking as well as significant event labels such as September 11th , Oklahoma City Bombing or USS Cole Bombing.

ASSESSMENT OF METHODOLOGICAL QUALITY Empirical studies will be determined first as those studies involving the analysis of empirical data as described above. These studies will then be individually examined to determine if the subject is relevant to the prevention, detection, response and management of terrorism events as described above. Each study will then be examined individually in terms of the methodological design used. Some guidance will be obtained from the Maryland Report (see

16

Sherman et al., 1997; Sherman et al., 2002) in terms of the studys rating on the Sc ientific Methods Scale (SMS). Moderately rigorous studies will be those who could be rated a 3 on the SMS scale, although we will evaluate each study on its own merits. Furthermore, we found Farringtons (2003) criteria to also be helpful in determining the scientific validity of studies and will use both guidance form the Sherman et al.s SMS and Farringtons discussion of validity.

CRITERIA FOR DETERMINATION OF INDEPENDENT FINDINGS We anticipate that an evaluation research study may include multiple findings on a variety of different types of strategies or on different study samples. Each research work will be reviewed in terms of whether outcomes arise from different samples, or if different outcomes arise from the same study samples. For those arising from independent samples but reported in one article or book, we will treat as separate findings. For multiple findings from the sa me sample, each will be scrutinized independently as to how to either combine the findings or choose one to represent the study.

DETAILS OF STUDY CODING CATEGORIES As already mentioned, in our preliminary analysis we collected basic information about the citation, author, title, year, and abstract of the study. We also coded studies generally as empirical, thought pieces or cases studies and labeled the source of each study and the subject matter covered. In the systematic review, we will also record the types of outcomes measured, sample sizes, type of methodological design, unit of analysis, statistical tests used, reports of statistical significance if any, effect sizes and power statistics (if any), type of intervention, and the location or venue of intervention. Furthermore, rationales for studies given by authors and

17

conclusions made by authors will also be recorded. Because the number of studies finally used in the systematic review is anticipated to be small, multiple authors of this review will separately analyze each study and come to a consensus about the coding of the data.

STATISTICAL PROCEDURES AND CONVENTIONS The results of each study will be summarized and presented in table format listing relevant variables such as findings, effect sizes, statistical significance, type of intervention and type of methodology used. If possible, meta-analytic approaches to combining data from studies will be used (see Lipsey and Wilson, 2001) . When possible, the authors will calculate effect sizes if enough data is present in the study results to do so. We plan to use commonly accepted standardized measures of effect sizes (see Lipsey and Wilson, 2001; Rosenthal, 1991) to calculate and compare effect sizes. Furthermore, a general assessment will be made as to whether positive, negative or null findings are found. This will provide a general assessment as to whether the study investigator and the reviewers saw the conclusion to have a positive effect, no detected effect or the effect was not statistically significant or the program or treatment had an unintended backfire effect (Weisburd et al., 2001: 58-59).

TIMEFRAME The timeframe for this review will be as follows:

October 2004 December 2004: Continuance of literature review as described above as well as preliminarily examining empirical studies for inclusion. January 2004 March 2005: Data extraction.

18

March 2005 April 2005: Preliminary analysis of data and possible preparation for presentation at professional conferences. April 2005 December 2005: Final completion of data analysis, write up and presentation for review.

PLANS FOR UPDATING THE REVIEW It is anticipated the in January of 2007, the primary investigator will initiate the process for updating this review.

ACKNOWLDEGMENTS We would like to acknowledge the invaluable assistance of Louise Stanton and Michele Grillo who assisted in the general review described in the Background section of this protocol. Furthermore, we acknowledge the institutions from which the authors are supported: Northeastern University and Rutgers University.

STATEMENT CONCERNING CONFLICT OF INTEREST No parties involved in this review have a conflict of interest at this time.

REFERENCES Arlacchi, P. (2001). Opening statement. Seminar on human security and science and technology (10 October 2001). Laxenburg. [On-line]. Available: http://www.unodc.org/unodc/speech_2001-10-10_1.html. Blumstein, A. (1988). Prison populations: A system out of control? In Michael Tonry and Norval Morris (eds.), Crime and Justice: A Review of Research Volume 10. Chicago:

19

University of Chicago Press. Blumstein, A. and Wallman, J. (eds.). (2000). The crime drop in America. Cambridge, UK: Cambridge University Press. Cohen, Stan. (1972). Folk Devils and Moral Panics: The Creation of the Mods and Rockers. Oxford: Blackwell. Farrall, S., Bannister, J., Ditton, J., and Gilchrist, E. (1997). Open and closed question. Social research update 17. University of Surrey. [On-line]. Available: http://www.soc.surrey.ac.uk/sru/SRU17.html. Farrington, David. (2003). Methodological Quality Standards for Evaluation Research. The Annals of the American Academy of Political and Social Science, Vol. 587(1) , 49-68. Gover, A., D. MacKenzie, G. Styve. (2000). Boot Camps and Traditional Correctional Facilities for Juveniles: A Comparison of the Participants, Daily Activities, and Environments. Journal of Criminal Justice 28(1): 53-68. Guinnessy, J., and Dawson. P. (2002). Terrorism drives Bush R&D money to defense and NIJ; The science funding flat in fiscal 2003. Physics Today, 55, 30. Halkides, M. (1995). How not to study terrorism. Peace Review, 7, 253-260. Hoffman, B. (1992). Current research on terrorism and low-intensity conflict. Studies in Conflict and Terrorism, 15, 25-37. Issues in Science and Technology (Report). (2002). Federal R&D in FY 2002 will have biggest percentage gain in 20 years. Issues in Science and Technology, 18, 23. Lane, J., and Meeker, J. (2000). Subcultural diversity and the fear of crime and gangs. Crime and Delinquency, 46, 497-521. Laniel, L. (1999). The relationship between research and drug policy in the United States. Discussion Paper 44. Management of Social Transformations Programme. Paris. [Online]. Available : http://www.unesco.org/most/dplaniel.htm#author. Lipsey, M. and D. Wilson. (2001). Practical meta-analysis. Applied Social Research Methods Series 49. Thousand Oaks, CA: Sage. Lipton, D., Martinson, R., and Wilks, J. (1975). The effectiveness of correctional treatment: A survey of treatment evaluation studies. New York: Praeger. Lum, C. and Yang, S. (2004). Why do evaluation researchers in crime and justice choose nonexperimental methods? Presentation at the Third Annual Jerry Lee Crime Prevention Symposium. University of Maryland, College Park.

20

Macilwain, C. (2002). Bush goes to war as budget boosts R&D. Nature, 415, 564. MacKenzie, D. (1997). Criminal justice and crime prevention. In Sherman, Lawrence, Denise Gottfredson, Doris MacKenzie, John Eck, Peter Reuter and Shawn Bushway (eds.) Preventing crime: What works, what doesnt, whats promising. Washington, DC: National Institute of Justice. MacKenzie, D. (2000). Evidence-Based Corrections: Identifying What Works. Crime and Delinquency 46:457-71. Miller, R. (1988). The literature of terrorism. Terrorism, 11, 63-87. Parachini, J. (1999). Combating terrorism: Assessing the threat. Testimony to the House Subcommittee on National Security, Veterans Affairs and International Relations (October 20, 1999). [On-line]. Available: http://lxmi.mi.infn.it/~landnet/Biosec/parachini.pdf. Reuter, P. (2001). Why does research have so little impact on drug policy? Addiction, 96, 373376. Romano, T. (1984). Terrorism: An analysis of the literature. Dissertation. Fordham University, Department of Sociology, Criminology and Penology. Rosenthal, R. (1991). Meta-Analytic Procedures for Social Research. Applied Social Research Methods Series 6. Newbury Park: Sage. Schmid, A. (1983). Political terrorism: A research guide to concepts, theories, databases, and literature. New Brunswick: Transaction Books. Schmid, A. P. and Jongman, A.J. (1988). Political Terrorism: A new guide to actors, authors, concepts, databases, theories and literature. Amsterdam: North-Holland Publishing Company. Sebba, L. (2001). On the relationship between criminological research and policy: The case of crime victims. Criminal Justice: The International Journal of Policy and Practice, 1, 2758. Sherman, L. (1992). Policing domestic violence: Experiments and dilemmas. New York: Free Press. Sherman, L.W., Farrington, D.P., Welsh, B.C. and MacKenzie, D.L. (Eds.), (2002). Evidence based crime prevention. London, UK: Routledge. Sherman, L.W., Gottfredson, D., MacKenzie, D.L., Eck, J., Reuter, P. and Bushway, S. (1997). Preventing Crime: What Works, What Doesnt, Whats Promising: A Report to the United States Congress. Washington, DC: National Institute of Justice.

21

Silke, Andrew (ed.). (2004). Research on Terrorism: Trends, Achievements and Failures. New York: Taylor and Francis. Walter, E.V. (1969). Terror and Resistance: A Study of Political Violence with Case Studies of Some Primitive African Communities. New York, NY: Oxford University Press. Weisburd, David, Cynthia M. Lum, and Anthony Petrosino. (2001). Does Research Design Affect Study Outcomes in Criminal Justice? Annals of the American Academy of Political and Social Science 578:50-70. Weisburd, David, Anthony Petrosino and Cynthia Lum (eds.). (2003). Assessing systematic evidence in crime and justice: Methodological concerns and empirical outcomes (Preface). The Annals of the American Academy of Political and Social Science, 587, 614.

22

TABLES AND FIGURES Figure 1. Yearly Distribution of Terrorism Publications (as a percentage of total N)
40% 35% 30% 25% 20% 15% 10% 5%
1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

0%

Year
All articles (N=14,006) Peer-reviewed, non criminal justice (N=5,797)

Figure 2. Methodological Quality of Terrorism Research

Case Studies 1%

Empirical Research 3%

Thought Pieces 96%

23

Table 1. Distribution of Subject Matter in Terrorism Research (Percentages) Peerreviewed sources (N=4,458a) 18.1%
b

Empirical only (N=156 a) 10.3% 5.1% 1.9% 18.1% 5.2% 3.9% 25.8% 3.9% 0.6% 18.7% 1.3% 0.6% 1.3% 1.3% 0.6% 0.6% 0.6%

Subject Matter Weapons of mass destruction (biological, chemical, nuclear) Article on a specific issue such as the IRA, Al Qaeda or incident Political responses to terrorism (war, politics, international relations) Causes, motivations, psychology, trends of terrorism Impacts of terrorism (political, social, economic) Non-political responses to terrorism(medical, social, economic) Victimology, coping mechanisms, psychological effects of terrorism Other (nationalism, intelligence issues, democracy and vulnerability) Legal issues surrounding terrorism The media and public attitudes towards terrorism How to define terrorism Non-conventional, cyber and narco- terrorism Religion and terrorism State-Sponsored terrorism Law enforcement responses to terrorism (airports, police) Research/science of studying terrorism Domestic terrorism
a b

12.2% 9.5% 8.7% 7.7% 5.5% 5.4% 5.4% 5.2% 4.6% 4.2% 3.0% 2.6% 2.6% 2.5% 2.1% 0.6%

Excluding book reviews and articles where not enough information was given to be categorized. If could not be placed into any other category.

24

APPENDIX A: Organizations Conducting Terrorism Research

ORGANIZATION Adolescents in Political Violence Project (University of Tennessee) ANSER Institute for Homeland Security Aon Corporation Belfer Center for Science and International Affairs (Harvard University) Bioterrorism Preparedness Office (Center for Disease Control) Board of Neuroscience and Behavioral Health (National Academy of Science) Homeland Security Project (Brookings Inst.) Canadian Security Intelligence Service (Canadian Government) Center for Defense and International Security Studies Center for Biosecurity and Public Health Preparedness (University of Texas Houston) Center for Civilian Biodefense Strategies (Johns Hopkins University) Center for Contemporary Conflict (US Navy) Center for Defense Information - Terrorism Project Center for Democracy and Technology Center for International Security and Cooperation (Stanford University) Center for Non-Proliferation Studies (Monterey Institute of International Studies) Center for Peace and Security (Georgetown University) Center for Public Health and Disasters (UCLA) Center for Technology and National Security Policy (US Department of Defense) Center for Terrorism Preparedness (University of Findlay) Center for the Prevention of Genocide (Improve the World International) Center for the Study of Bioterrorism and Emerging Infections (St. Louis University) Center for the Study of Public Security (Rutgers University)

URL http://cfs.utk.edu/f_s/barber.html http://www.homelandsecurity.org http://www.aon.com/ http://bcsia.ksg.harvard.edu/ http://www.cdc.gov/ http://www.iom.edu http://www.brookings.edu/fp/research/projects/homeland/homel and.htm http://www.csis -scrs.gc.ca/eng/menu/welcome_e.html http://www.cdiss.org/terror.htm http://www.sph.uth.tmc.edu/cbphp http://www.hopkins-biodefense.org http://www.ccc.nps.navy.mil/ http://www.cdi.org http://www.cdt.org/ http://cisac.stanford.edu http://cns.miis.edu/ http://cpass.georgetown.edu/ http://www.cphd.ucla.edu http://www.ndu.edu/ctnsp/index.html http://www.nceem.org/terrorism/default.asp http://www.genocideprevention.org/index.htm http://bioterrorism.slu.edu/ http://www.andromeda.rutgers.edu/~rcst/home.html

25

Center for the Study of Terrorism and Political Violence (University of St. Andrews) Center on Terrorism and Irregular Warfare (US Navy) Center on Terrorism and Public Health (Florida State College of Medicine) Center on Terrorism and Public Safety (John Jay College of Criminal Justice) Centers for Public Health Preparedness (Center for Disease Control) CERT Coordination Center (Carnegie Mellon University) Chemical and Biological Arms Control Institute Chemical and Biological Arms Control Program (Federation of American Scientists) Chemical and Biological Defense Information Analysis Center (Battelle Memorial Institute) Columbia University World Trade Center Archive Project Command and Control Research Program (Department of Defense) Conflict Archive on the Internet (University of Ulster) Congressional Research Service (Library of Congress) Council on Foreign Relations Counterterrorism Office (Department of State) Critical Incident Analysis Group (University of Virginia) Dart Center for Journalism and Trauma (University of Washington) Defense Advanced Research Projects Agency (Department of Defense) Defense Information Systems Agency (Department of Defense) Defense Intelligence Agency (Department of Defense)

http://www.st-andrews.ac.uk/academic/intrel/research/cstpv/ http://www.nps.navy.mil/ctiw/ http://www.med.fsu.edu/healthaffairs/ctph/default.asp http://www.centeronterrorism.org http://www.phppo.cdc.gov/owpp/CPHPLocations.asp http://www.cert.org/ http://www.cbaci.org/ http://www.fas.org/bwc/index.htm http://www.cbiac.apgea.army.mil/ http://www.columbia.edu/cu/lweb/news/libraries/2001-1030.wtc_archives.html http://www.dodccrp.org/ http://cain.ulster.ac.uk/ http://www.loc.gov/crsinfo/whatscrs.html#about http://www.cfr.org/ http://www.state.gov/s/ct/ http://www.healthsystem.virginia.edu/ciag http://www.dartcenter.org/ http://www.darpa.mil/ http://www.disa.mil/ http://www.dia.mil/

Emergency Response and Research Institute Facts on File, Inc (Ferguson Publishing Co.) Federal Research Division-Terrorism and Crime Studies (Library of Congress) Financial Action Task Force on Money Laundering Foreign Policy Research Institute

http://www.emergency.com http://www.factsonfile.com/ http://lcweb.loc.gov/rr/frd/ http://www1.oecd.org/fatf/index.htm http://www.fpri.org/

Foundation for the Defense of Democracies 26

http://www.defenddemocracy.org/

National Security Archive - George Washington University http://www.gwu.edu/~nsarchiv/ Government Accounting Office- Special Collections- Terrorism Henry L. Stimson Center Human Rights Watch Institute for Security Technology Studies (Dartmouth College) International Center for Terrorism Studies (Potomac Institute for Policy Studies) International Critical Incident Stress Foundation International Policy Institute for Counter-Terrorism International Society for Traumatic Stress Studies Jaffe Center for Strategic Studies at Tel Aviv University Library of Congress September 11 Archive http://www.gao.gov/terrorism.html http://www.stimson.org http://www.hrw.org/ http://www.ists.dartmouth.edu/ http://www.potomacinstitute.org/academic/icts.cfm http://www.icisf.org/ http://www.ict.org.il/ www.istss.org http://www.tau.ac.il/jcss/about.html http://september11.archive.org/ http://www.mackenzieinstitute.com/ http://www.gspia.pitt.edu/ridgway/ http://www.mitretek.org/home.nsf http://www.narcoterror.org/ http://www.naic.org/ http://www.nima.mil/ http://www.nsa.gov/ http://www.ntis.gov http://www.nci.org/ http://www.customs.ustreas.gov/ http://www.treasury.gov/offices/domestic-finance/financialinstitution/terrorism -insurance/ http://www.ustreas.gov/offices/enforcement/ofac/ http://www.mipt.org/ http://www.pnl.gov/ http://www.dartmouth.edu http://pgis.pinkertons.com/

The Mackenzie Institute

Matthew B. Ridgway Center for International Security Studies (University of Pittsburgh) Mitretek Systems Narcoterror.org National Association of Insurance Commissioners National Geospatial Intelligence Agency (formerly National Imagery and Mapping Agency, Department of Defense) National Security Agency National Technical Information Service (Department of Commerce) Nuclear Control Institute US Customs & Border Protection (DHS) Office of Domestic FinanceTerrorism Risk Insurance Program (Department of Treasury) Office of Foreign Asset Control (Department of Treasury) Oklahoma City National Memorial Institute for the Prevention of Terrorism Pacific Northwest National Laboratory (Department of Energy) PILOTS catalog Dartmouth College Library Pinkerton Global Intelligence Services

27

Political Risk and Trade Credit Group (MMC Enterprise Risk Group) Public Health Practice Program (Center for Disease Control) RAND SAPRA India South Asia Terrorism Portal (Institute for Conflict Management) SSAF International Ltd. Technical Support Working Group (Department of State) Technology and Public Policy Program (Center for Strategic and International Studies) Global Programme Against Terrorism (United Nations) Terrorism Research Center The Cato Institute The New York Times The Washington Post Transnational Threats Initiative (Center for Strategic and International Studies) United States Institute of Peace US Department of Agriculture Food Safety & Inspection Service; Agricultural Research Service US Food and Drug AdministrationCenter for Biologics Evaluation & Research; and Center for Food Safety and Applied Nutrition US Secret Service US Air Force Counter Proliferation Center Versar Inc. Wade Financial Group William R. Nelson Institute for Public Affairs (James Madison University)

http://www.mmcer.com/ http://www.phppo.cdc.gov http://www.rand.org/ http://www.subcontinent.com/sapra.html http://www.satp.org/ http://www.ssafprotection.com/ http://tswg.gov http://www.csis.org/tech/index.htm http://www.odccp.org/odccp/terrorism.html http://www.homelandsecurity.com/ http://www.cato.org/ http://www.nytimes.com/ http://www.washingtonpost.com/ http://www.csis.org/tnt/ http://www.usip.org/ http://www.fsis.usda.gov/; http://www.ars.usda.gov/ http://www.fda.gov/cber/index.html; http://vm.cfsan.fda.gov/list.html http://www.secretservice.gov/index.shtml http://www.au.af.mil/au/awc/awcgate/awc-cps.htm http://c21.maxwell.af.mil/ http://www.versar.com/ http://www.terrorism-insurance-risk-management.com/ http://www.jmu.edu/orgs/wrni/

28

APPENDIX B: Data Sources Available

ORGANIZATION DATA DESCRIPTION http://www.yale.edu/lawweb/avalon/terrorism/terror.htm documents pre -18th century Avalon Project, Yale Law School CDISS Terrorism Program - Center for Defense and International Security Studies Center for the Prevention of Genocide Center for Non-Proliferation Studies Center for the Study of Terrorism and Political Violence- University of St. Andrews Center on Terrorism and Irregular Warfare Chemical and Biological Defense Information Analysis Center Columbia University World Trade Center Archive Project Conflict Archive on the Internet Counterterrorism Office- US Department of State Department of Psychology-Political Violence Program- Tel Aviv University Emergency Response and Research Institute Federal Research Division-Terrorism Studies Terrorist incidents 1945 to 1998 Country reports online Weapons of mass destruction database Terrorism incident database Government databases. Suicide bombers. CBIAC Bibliographic Database Archive data on September 11th Databases on Northern Ireland conflict Patterns and trends of terrorism, chronology online Ariel Merari database on terrorism incidents Privately-held databases on terrorist personnel, terrorism methods, terrorist incidents, and the implications thereof Terrorism databases http://www.fas.org/irp/world/para/index.html This directory of para-states is not a list of terrorist organizations, and is not constructed to supplement or complement the list of terrorist organizations of the US Department of State. The guide intentionally casts a wide net, and includes both the nasty and nice.

Federation of American Scientists (FAS), Intelligence Resource Program, Liberation Movements, Terrorist Organizations, Substance Cartels, and other Para-State Organizations. George Washington University

September 11 Source Books

29

Henry L. Stimson Center International Center for Political Violence and Terrorism Research (ICPVTR)

Unconventional weapons, chemical and biological terrorism http://www.ntu.edu.sg/idss/research_03a.htm The ICPVTR database focuses on the Asia Pacific region, especially Southeast Asia. No other details are available and it is not yet available to the public.

International Policy Institute for Counter-Terrorism (ICT) Israel Library of Congress University of Michigan Documents Center Americas War Against TerrorismWorld Trade Center/Pentagon Terrorism and the Aftermath

Houses six databases on terrorists groups, incidents, activity Archive of digital materials up after 9/11

http://www.lib.umich.edu/govdocs/usterror.html Insurance related databases such as claims from attacks and other criminal liabilities News archives RAND Terrorism Chronology Database, RAND-MIPT Terrorism Incident Database, and MIPT Indictment Database Published International Literature On Traumatic Stress Incident Database (currently housed and researched at the University of Maryland, Department of Criminology under Professor Gary LaFree) RAND Terrorism Incident Database Multiple databases

National Association of Insurance Commissioners The New York Times Oklahoma City National Memorial Institute for the Prevention of Terrorism PILOTS catalog

Pinkerton Global Intelligence Services RAND SSAF International Ltd Technical Support Working Group (US Department of State) Terrorism Prevention Branch- Center for International Crime Prevention (CICP) of the UN Center for Non-Proliferation Studies Terrorism Research Center Wade Financial Group

10 databases described in Forum on Crime and Society Terrorist Group Profiles Terrorist group profiles, significant events, terrorist attacks, counterterrorism group profiles, and country profiles. Insurance related information

30

The Washington Post

News archives

ENDNOTES
1

We also conducted a search from approximately twenty different university libraries for books related to terrorism. However, because of the inconsistent availability of an abstract for each citation, an analysis of the books was not undertaken at this time. However, the systematic review will include evaluations from all known mediums and sources. 2 The databases used were Academic Search Premier, ArticleFirst (OCLC), Contemporary Womens Issues, Criminal Justice Abstracts, EbscoHost, EconLit, Educational Abstracts, Electronic Collections Online, ERIC(OCLC), GEOBASE, Humanities Abstracts, Ingenta, ISI Web of Science, MEDLINE, National Criminal Justice Reference Service, PAIS International Articles Only, PUBMEDLINE, Social Science Abstracts, Sociological Abstracts. The time periods covered by each of these databases can be obtained at http://www.lib.neu.edu/gateway/databasestrifold.pdf. 3 Although an attempt was made to eliminate duplicates using database manipulation software, it is possible that not all duplicates were initially eliminated. Because this review later focused on peer-reviewed literature only, each of these were individually checked by hand for the existence of duplicates which were then excluded. The 14,006 articles represent the final universe of articles after this check was made on the peer-reviewed articles only. 4 See http://www.epnet.com/titlelists.asp . 5 See http://www.pais.org/journalslist/peer-reviewed.stm . 6 We used, at the time, http://www.ama-assn.org/med_link/peer.htm which is now an inactive link. , 7 See http://www.eurekalert.org/links.php?jrnl=A . 8 H.R. 3162, 107th Congress, 1st Session, October 24, 2001. 9 This search was conducted by Louise Stanton at Rutgers University.

31