NEWS FEATURE

NATURE|Vol 465|27 May 2010 Vol

Science for the masses

left, centre: k. genry; right: g. m. zenner petersen

The US National Science Foundations insistence that every research project addresses broader impacts leaves many researchers baffled. Corie Lok takes a looks at the system.
esearch-funding agencies are forever trying to balance two opposing forces: scientists desire to be left alone to do their research, and societys demand to see a return on its investment. The European Commission, for example, has tried to strike that balance over the past decade by considering social effects when reviewing proposals under its various Framework programmes for research. And the Higher Education Funding Council for England announced last year that, starting in 2013, research will be assessed partly on its demonstrable benefits to the economy, society or culture. But no agency has gone as far as the US National Science Foundation (NSF), which will not even consider a proposal unless it explicitly includes activities to demonstrate the projects broader impacts on science or society at large. The criterion was established to get scientists out of their ivory towers and connect them to society, explains Arden Bement, director of the NSF in Arlington, Virginia. Unfortunately, good intentions are not enough to guarantee success, says Diandra Leslie-Pelecky, a physicist at the University of Texas in Dallas who is active in popular science writing and other forms of outreach. Leslie-Pelecky remembers a pilot project she carried out in 2001, when she was at the University of Nebraska in Lincoln. In many ways, it was typical of the kinds of things that
416

NSF-funded researchers do to fulfil their activities listed in the foundations proposal broader-impacts requirement. She took three guide are: developing educational materifemale graduate students on weekly visits to als for elementary, high-school and underlocal classrooms, where they spent 45-min- graduate students; involving these students utes leading nine- and ten-year-old children in the research where appropriate; creating in practical activities designed to teach them mentoring programmes; maintaining and about electricity and circuits. The visitors operating shared research infrastructure; also talked about their lab work and careers. presenting research results to non-scientific In addition, Leslie-Pelecky audiences such as policy-makdid something less typical ers; establishing international, The criterion was of broader-impacts efforts: industrial or government established to get she brought along education collaborations; developing researchers to study the effect exhibits in partnership with scientists out of of this interaction on the chilmuseums; forming start-up their ivory towers drens perception of scientists. companies; and giving presand connect them Those assessments were entations to the public. startling, she says. After three Because it lacks conceptual to society. months, most of the students clarity, the broader-impacts said that they still werent sure requirement often leaves who these young teachers were except that researchers unsure about what to include they couldnt possibly be scientists. In their in their proposals, and leads to inconsistenminds, scientists were unfriendly, grey-haired cies in how reviewers evaluate applications. old men in white lab coats1. Broader impacts were designed to be open, And thats what I worry about with broader but openness confuses a lot of people, says impacts, says Leslie-Pelecky. There are a lot of Luis Echegoyen, the division director for NSF people putting time and effort into [these sorts chemistry. of activities] and they have no idea if theyre To make matters worse, the NSF has made making any difference or not. little attempt to systematically track how its Many NSF-funded researchers find the broader-impacts requirements are being foundations definition of broader impacts met, or how much grant money is being to be, perhaps unsurprisingly, broad, and spent in the process. Nor does it have a sysfrustratingly vague. Among the examples of tem in place to evaluate the effectiveness of
2010 Macmillan Publishers Limited. All rights reserved

NATURE|Vol 465|27 May 2010 Vol 465|27 May 2010

NEWS FEATURE

the various projects. These problems with the broader-impacts requirement have been confirmed over the past decade in studies from the National Academy of Public Administration and elsewhere. In March, the NSFs oversight body, the National Science Board, launched a task force to examine how broader impacts can be improved. Chaired by Alan Leshner, chief executive of the American Association for the Advancement of Science in Washington DC, the task force is not expected to make its recommendations until 2011. In the meantime, a small number of academic institutions are already exploring ways to make broaderimpacts efforts work better. After all, says Ralph Nuzzo, a chemist and materials scientist at the University of Illinois in UrbanaChampaign, most US scientists have come to accept even if grudgingly that it is probably a good idea to demonstrate the wider implications of their work. People want to do the right thing, says Nuzzo. Its just hard to know what that is.

about how their work is connected to enhanc- researchers to maximize the commercial effect ing benefits to society, says Robert Mathieu, of their research. chair of the astronomy department at the UniA preliminary network for broader impacts versity of WisconsinMadison and director already exists. Stanford, for example, has an of the universitys NSF-funded Center for the Office of Science Outreach, which helped Integration of Research, Teaching, and Learn- Spakowitz to make the initial contacts ing (CIRTL). As an example, Mathieu points to to get his project started at the hospital. the NSFs prestigious CAREER award for junior And Mathieus centre at the University of faculty members, which requires that applicants WisconsinMadison is part of a network propose educational activities, such as design- of six CIRTLs located at research campuses ing courses and carrying out public-outreach such as Vanderbilt University in Nashville, activities, that are integrated with the proposed Tennessee, and Texas A&M University in research. He has sat on several College Station. The Wisconreview panels, and says that the sin centre runs workshops It makes scientists education section of proposals individual think more explicitly and conductswith faculty has grown in length and sophisconsultations about how their tication over the years. members needing assistance But despite the NSFs efforts work is connected to with integrating broaderto educate scientists about impacts activities into their enhancing benefits broader impacts through grant proposals. The other to society. websites, workshops and CIRTLs are moving towards conference sessions, most still similar sorts of programmes. approach the criterion with confusion and Mathieu and his group are putting together dread. Researchers often end up repackaging plans to expand this network to 2025 uniwhat theyre already doing. Overwhelmingly, versities. Their ultimate goal is for any US Scientists get creative says Echegoyen, the number one broader research university that wants its own CIRTL Nonetheless, there have been some successes. impact that most people in the chemistry to have one, creating a community that shares At Rensselaer Polytechnic Institute in Troy, division are using is training graduate students best practices among its researchers and New York, for example, the NSFs Nanoscale and postdocs. other professionals, and develops the experScience and Engineering Center for Directed One problem is that the kind of support tise to effectively broaden impacts. Mathieu Assembly of Nanostructures sponsors the network that researchers take for granted estimates that establishing CIRTLs at the Molecularium project, which has produced working with collaborators, sharing ideas and nations top research universities would cost teachers materials on nanoscience and an advice, learning from published results, attend- roughly US$100 million over five years. animated three-dimensional IMAX film ing conferences is still rudimentary when Yet such ideas lead to a more fundamental called Molecules to the Max. At the University it comes to broader impacts. A useful model question. Is having every principal investigator of WisconsinMadison, microbial biochemist could be the network of technology-transfer working individually on broader impacts for Douglas Weibel and his group have prepared a offices that are found on many US campuses, which many are inexperienced and untrained child-friendly, interactive display about micro- which have been instrumental in helping the most efficient way of achieving the maxscopy that they exhibit every year imum effect? at the universitys one-day public Some scholars say no. science exposition. At Stanford In a paper published last University in California, chemiyear, Warren Burggren, a cal engineer Andrew Spakowitz biologist and dean of the spends two to three hours a College of Arts and Sciweek working with graduate and ences at the University undergraduate students to proof North Texas in Denvide workshops for patients at ton, writes that the job of Stanfords Lucile Packard Chilimplementing broader drens Hospital, most of whom impacts should fall to the are unable to attend school. researchers institution, Spakowitz and his group crenot to the researcher him ate the workshops that cover or herself 2 . The institutopics such as pH and gravity, tion, be it college, departand lead the hands-on activiment or centre, would pool ties at the hospital. a portion of the NSF grants Some say that the broaderobtained by its members impacts criterion has helped and hire the professionals to catalyse a change in the needed to broaden impacts research-focused culture of effectively. Scientists should academic science. It makes Stanford chemical engineer Andrew Spakowitz (right) teaches children at the Lucile still be involved, but the scientists think more explicitly Packard Childrens Hospital as part of his NSF award. coordination would happen
2010 Macmillan Publishers Limited. All rights reserved

417

k. ho, lucile packard childrens hospital

NEWS FEATURE

NATURE|Vol 465|27 May 2010 Vol

at the institutional level. I think it will be more efficient, because youve got people doing what theyre trained for, says Burggren. Another idea, suggested by Barry Bozeman, a sciencepolicy expert at the University of Georgia in Athens, is for the NSF to create specific research programmes with strong broader-impact goals around areas in which the effects are important and obvious, such as climate change3. Bozeman says that the NSF is already following this strategy with awards that, for example, Outreach activities such as a board game (left) about the link between nanotechnology and society, and biochemist promote the recruitment and Douglas Weibel (right) presenting a hands-on microscope display to children, can help bring science to the public. retention of women in academic science. yielded some results, but also showed that perhaps reflects more fundamental issues The NSFs broader-impacts requirement the research and the broader-impacts work about the relationship between science and took its current form in 1997, when the founda- were often so interwoven that it was difficult society, says Britt Holbrook, a philosopher of tion simplified the criteria used by reviewers to to tease them apart. science at the University of North Texas. Is the evaluate proposals. Two of the four criteria Mostly, evaluation happens as a by-product NSF passing the buck by asking scientists to the intrinsic scientific merit of the project, and of other NSF activities routine reviews of meet what is essentially a political goal: demthe soundness of the teams approach were grantees annual reports, for example, or the onstrating the benefits of science? merged into one, known as intellectual merit. regular review of programmes at each diviMy hypothesis is that the NSF has passed And the other two the utility or relevance sion carried out by a committee of external some of that burden to the people getting of the project, and its effect on the infrastruc- scientists. funded, says Holbrook, who has a grant ture of science and engineering By not tracking broader- from the foundation to study how different were collapsed into broader impacts activities, the NSF funding agencies incorporate societal impacts By not tracking impacts. undervalues its true contribution into their review process. But when you do broader-impacts For the first few years, many to society, says Melanie Rob- that, you get push back from the scientific activities, the NSF proposers and reviewers ignored erts, an assistant director at the and engineering community because it goes this second criterion, treating it Colorado Initiative in Molecular against the traditional idea of peer review, undervalues its with the same disregard that Biotechnology at the University which is designed to assess work at a technitrue contribution they had previously expressed of Colorado in Boulder who has cal or scientific level. to society. towards utility or relevance. It analysed the broader-impacts And how does the NSF show impact, given was only in 2002 that the NSF statements from recent grant that the agencys specialty, basic research cracked down, announcing that any proposal abstracts. It is missing an opportunity to create often doesnt have an immediate pay-off, or that didnt separately address both the intel- a knowledge base of how to carry out broader- else has a pay-off that is difficult to quantify? lectual-merit and the broader-impacts criteria impacts activities effectively and reward those Its a delicate balancing act, says Neal Lane, a would be returned without review. who do a good job. physicist at Rice University in Houston, Texas, The confusion that persists despite the NSFs who was the NSF director from 1993 to 1998 The right track repeated attempts to clarify broader impacts when the broader-impacts criterion was first Of the few small-scale efforts to track and implemented. Its important to get scientists to assess the broader-impacts requirement, none think about how their work affects society, he has been conclusive. In 2008, for example, says. But one has to be careful not to push it too the NSF sent Congress a report on broader far. If the NSF moves too far in the direction of impacts, as mandated in the 2007 America doing things that have short-term benefits, then COMPETES Act but included little more I think that is not consistent with the NSFs misthan anecdotal descriptions of specific sion, and that would not be good for American research projects. The chemistry division science, engineering and technology. recently contracted with an outside comCorie Lok is an editor for Nature in Boston, pany to assess the broader-impacts activities Massachusetts. of a sample of its grantees, but the project 1. Buck, g. a. et al. J. Elem. Sci. Educ. 14, 110 (2002). fell into limbo when the company dissolved. 2. Burggren, W. W. Soc. Epistemol. 23, 221 (2009). And the geosciences directorate carried out 3. Bozeman, B. & Boardman, c. Soc. Epistemol. 23, 183 (2009). informal surveys of broader-impacts activi- The film Molecules to the Max was created by an ties in the ocean and Earth sciences, which NSF-funded nanoscience centre. See Editorial, page 398.
418
2010 Macmillan Publishers Limited. All rights reserved

rensselaer polytech. inst.

r. nygard; t. dassler

Copyright of Nature is the property of Nature Publishing Group and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.