"Science reveals new world's to explore, and by implication new opportunities to seize and new futures to create." This statement by Vice President Al Gore,[Note 16] quoted in Science in the National Interest, captures the importance of a strong National investment in science and technology. For the Department of Energy and its laboratories, science and technology are the currency for meeting our mission requirements. The Laboratories support world-class scientists and engineers and unique, advanced research facilities which, help address complex, multi-disciplinary problems in areas ranging from national security to fusion energy to environmental clean-up.
The research facilities at the Laboratories provide access for thousands of academic and industrial scientists to new frontiers in areas such as materials science and molecular biology. In this fashion, and by virtue of their distinguished record of scientific accomplishments, the Laboratories represent a National asset that warrants careful stewardship during an era when science holds the potential for addressing major National needs in health care, environmental quality, national security, and sustainable development. The Administration's new science policy provides the framework for helping sustain and guide the Department's scientific facilities and programs in the face of tight competition for resources. These facilities, like basic science in general, have provided a means of discovery and a record of technological innovation. The dividends of this investment will continue to accrue for generations to come.
Merit review with peer evaluation is a powerful and effective tool for enhancing relevance and productivity in Federal research and development (R&D). Despite some of its well-documented shortcomings [Note 17], peer review stimulates competition, establishes high standards for quality, rewards productivity, and, on balance, fosters creativity and promotes fair play. When combined with energetic and visionary R&D program leadership, peer review can marshal highly competent R&D teams, focus scarce resources on the most important and potentially fruitful technical opportunities, and provide reasonable assurances to taxpayers that their Federal R&D dollars are being prudently invested.
On May 6, 1994, in a White House memorandum, the Office of Science and Technology Policy (OSTP) and the Office of Management and Budget (OMB) established merit review with peer evaluation as an "R&D policy principle" to be incorporated in all Federal agency R&D budgets for Fiscal Year 1996. Specifically, according to the memorandum, each Federal R&D agency is expected to
Further, increasing concern about accountability for efficient and productive use of government funds, including funds for government-supported R&D, has been reflected in recent Federal legislation and executive direction. The Chief Financial Officers Act, the Government Performance and Results Act, the Competition in Contracting Act, the revision of the Federal Acquisition Regulation, the National Performance Review initiative, and a number of other program evaluation initiatives from OMB have all had a profound effect on Federal agency management, oversight, and conduct of R&D programs.
The Department of Energy (DOE) fully embraces these principles of accountability, competition, and objective merit review, including peer review. In fact, it has already put in place many new ways of doing business that are strengthening their application. This paper documents the Department's continuing and expanding commitment to these principles and, in particular, to peer review.
At the Department of Energy, peer review means competent, qualified, objective, and formal evaluation using (1) specified criteria and (2) the review and advice of qualified peers. To be qualified, peers must be technically competent in the scientific and technical field under review. Peers may come from any source, including industry, academia, and government agencies and associated laboratories. To be objective, peers must be reasonably independent and free from conflict of interest. The results of peer reviews must be recorded and, under appropriate controls, accountable to further review.
Merit reviews meeting these criteria take on many and diverse forms. They are applied to R&D proposals, projects, and programs. They are applied, as well, to the design and acquisition of major research facilities and to the formulation of multiyear research plans and strategies. Appropriate forms of peer review are constructed and applied to activities at various organizational levels: the Department Secretary, Assistant Secretaries, program offices, National Laboratories, integrated laboratory R&D, research subcontractors (including universities), and laboratory user facilities. The nature of peer review at each level is tailored to the needs at that level.
Peer reviews are usually undertaken in the context of the allocation and use of scarce R&D resources. They may be used in conjunction with competitive selection processes, where peer reviews take place prior to the award or approval of a grant or contract, or where the research activities are chosen from a pool of qualified applicants following peer reviews. These types of peer review are called pre-award, or prospective, reviews. Peer reviews may also be used in conjunction with evaluations of ongoing or recently completed research. These in-progress or performance reviews are called post-award, or retrospective, reviews. These latter reviews also strongly influence the allocation of R&D resources by what is sometimes referred to as selection by competitive survival.
Although the terms prospective and retrospective are useful constructs to describe when merit reviews with peer evaluation take place, the substance of both types of reviews are quite similar. In both cases, the merit of an investigator's or research group's record of accomplishments (retrospective considerations) and the projected course of future research (prospective considerations) bear directly on the evaluation.
The Department of Energy, like other Federal R&D agencies, must carry out its scientific and technical missions within a larger context of statutory, regulatory, and procedural requirements governing the expenditure of R&D funds. This context varies for different programs, but in each case largely determines the way in which peer review principles and methods are applied.
The award of research contracts, for example, is governed by the Federal Acquisition Regulation and the Competition in Contracting Act, both of which require competition among bidders and formal selection processes. The Department employs peer review principles and methods, including the use of independent engineering and scientific reviewers, in the technical evaluation stage of all such selection processes related to R&D, except in relatively rare instances where sole-source selection may be justified.
Further, the award of research grants and cooperative agreements is governed by the Department's Financial Assistance Rules, as promulgated in the Code of Federal Regulations (10 CFR Part 600). The Department's major research organizations have promulgated formal rules in the CFR governing the merit review process for R&D financial assistance. These rules require the use of technical experts to perform credible merit reviews of all applications, solicited and unsolicited. Such merit reviews may make use of standing committees, ad hoc committees, or field readers, and generally include, in the spirit of peer evaluation, at least three qualified persons from outside the awarding program office, in addition to the designated contracting officer's representative.
A combination of Federal and Departmental regulations also governs the award of contracts at the Department's laboratories. Under the Federal Acquisition Regulation, a management and operating (M&O) contract is recognized as an appropriate instrument, or agreement, under which the government
Such M&O contracts permit the Department to draw upon, nurture, and maintain the special technical expertise and capabilities required for unique missions, such as those associated with nuclear weapons and large, multidisciplinary, integrated, non-weapons research. Over the years, the Department's missions and associated requirements for such specialized expertise and capabilities have given rise to the Department's laboratory system. Altogether, the replacement cost of the facilities of this system is currently estimated to exceed $30 billion. The laboratories employ about 50,000 people, representing a concentration of technical talent that includes more than 8,500 Ph.D.s and several Nobel laureates.
Examples of specialized research facilities located at these laboratories include accelerators for the study of high energy physics, the world's most powerful computers and lasers, synchrotron light sources for probing the structure of materials, facilities for producing medical isotopes, and instrumentation laboratories for characterizing the details of flame propagation and combustion. The Department owns and maintains these facilities and, with the exception of the classified facilities, makes them available to researchers from all sectors of the economy, public and private. The Department underwrites the operating costs for experimenters who openly share their data with the scientific community. Commercial users may also use the facilities to conduct proprietary research, but on the condition that they participate on a full-cost-recovery basis. Peer review is routinely employed to allocate available time and select the experiments conducted at the major research facilities, with some facilities having waiting lists exceeding a year.
Under a DOE-initiated contract reform, [Note 18] the Department's M&O contracts now require, or will soon require, regular performance-based merit reviews to ensure accountability in M&O contractor performance. M&O contracts that do not now contain such requirements will incorporate them when the contracts come up for renewal or renegotiation. In addition, all laboratories have an array of outside advisory panels that periodically review and advise on the relevance and productivity of laboratory-conducted R&D.
Finally, one M&O contractor seldom performs all of its R&D tasking by itself. Whether under a lead-laboratory or other management arrangement with the Department, a portion of the R&D is typically subcontracted to universities, private laboratories, or other R&D performers. At the National Renewable Energy Laboratory, for example, one-half of the laboratory's total funding supports research subcontracted to outside R&D performers. At Argonne National Laboratory, Oak Ridge National Laboratory, and Pacific Northwest Laboratory, this figure varies between 10 and 20 percent. At other laboratories, this figure is less. All such subcontracts, likewise, are governed by contract provisions that generally require both competitive selection processes, which in the case of R&D generally involve merit reviews with peer evaluation, and periodic evaluations of contractor performance.
The Department's overall R&D budget for Fiscal Year 1994 is estimated, depending upon one's precise definition of R&D, to be about $7.4 billion, as shown in Appendix A. This amount may be grouped into three broad, roughly equal, categories: fundamental science and energy research ($2.4 billion); civilian energy technology and related R&D ($2.8 billion); and national security R&D ($2.2 billion).
Of the $7.4 billion total, approximately 20 percent supports research carried out by R&D performers employed outside the Department and its laboratory system. Performers include industry, universities, public and private research institutions, and R&D consortia. The instruments used to convey funding to these R&D performers include Department-awarded grants, cooperative agreements and contracts, and laboratory-awarded research subcontracts. Of the remaining 80 percent, most supports research and related activities carried out by performers within the Department and its laboratory system. Of this, approximately 40 percent supports the operation, maintenance, construction, and modernization of the specialized research and related user facilities. Another 35 percent supports internal laboratory research programs. The remaining 25 percent supports other functions, including general infrastructure (for example, roads, utilities), overhead, and other indirect costs.
The mix of R&D activities calls for a variety of approaches to managing research and applying peer review principles and methods. For example, research by outside R&D performers, because of the nature of the procurement instruments used to convey funding, is governed by statutory and regulatory requirements that require, in one form or another, merit reviews, mostly with peer evaluation, in conjunction with pre-award competitive selection processes. The M&O contracts are, likewise, competed and regularly evaluated, with increasing emphasis on specific performance-based measurement criteria. Also, because experimental time on the special facilities is so highly valued and demand exceeds supply, virtually all access to the facilities is allocated through some means of merit review with peer evaluation.
Peer review coverage of the internal research programs at each laboratory is, likewise, varied. The greater portion is subject to retrospective merit reviews, called for by management and conducted most often by scientists who are independent of the laboratory, in conjunction with outside program reviews and advisory committee oversight. A lesser portion is subject to prospective peer review as exemplified somewhat narrowly by the highly successful laboratory directors' discretionary R&D program and more broadly by the many programs managed within Departmental headquarters that apply peer review principles and methods to the evaluation of laboratory Field Work Proposals. This latter process is illuminated later in this paper.
Even though the Department applies different peer review methods to guide its research programs, both outside and internal, a sampling of R&D projects, using retrospective merit review by independent experts, provides evidence that research quality and relevance of both types of research programs are comparable. For example, an organization within the Office of Energy Research (the Office of Program Analysis) regularly conducts, at the invitation of R&D program managers, retrospective peer reviews of R&D programs throughout the Department. Using an interactive method with independent, outside expert reviewers, this organization has evaluated more than 2,700 research projects over 12 years, covering about 20 percent of the Department's civilian basic research and technology development programs. The most recent data, which includes 744 research projects in Basic Energy Sciences conducted at both national laboratories and universities, produced results showing that the research programs of both internal and outside R&D performers shared nearly identical statistical profiles on research quality and relevance. These retrospective peer reviews, it should be noted, are in addition to other reviews administered by the program managers and serve as an independent measure of research quality and relevance.
Finally, above the project level, at higher levels of decisionmaking in the organizational hierarchy, the Department makes extensive, although not comprehensive, use of expert advisory bodies, constituted under the Federal Advisory Committee Act, and the National Academies. Peer input is also obtained from workshops, technical society meetings and symposia, and extensive publication in the peer-reviewed literature.
The scientific and technology development missions of the Department of Energy are extraordinarily diverse and far-ranging. The Department is among the largest supporters of fundamental science and basic research across many disciplinary areas and technical fields. Its applied research and technology development programs concentrate primarily on the Department's energy, environmental, and national security missions, but in doing so embrace countless forefront areas of research vital to industry, commerce, and trade. The Department also builds and equips many of the premier R&D facilities vital to U.S. competitiveness and used by U.S. universities, corporations, and nonprofit research institutes.
In these respects, the Department is endowed with highly valuable R&D resources for which there is intense competition. The Department has found over the years that this competition is most productively and equitably managed by merit review practices that involve objective reviews and advice, that is, by peer review. It has also found, however, that peer review practices must be appropriately tailored to each context, depending on the nature of the research activities performed and the R&D community served.
Finally, and importantly, peer review systems at the Department do not now, nor must they in the future, preclude the possibility of initiating some research programs without peer review. Preserving this flexibility is vital. Programs representing entirely new research directions, research at the interfaces between established communities, or essential elements in critical mission areas often do not survive traditional peer review. If the Department had applied peer review rigidly, without flexibility or regard to such weaknesses, it might not have funded Dr. Luis Alvarez, whose work ultimately led to the meteor-impact theory of the extinction of the dinosaurs. This was world-class science -- neutron activation analysis of iridium anomalies in soil samples at the Cretaceous-Tertiary geologic boundary -- that used the Department's skills and facilities in novel ways that led to a revolution in thinking about our planet and its history.
Virtually all of the Department's fundamental science and energy research programs undergo merit review of one form or another in order to ensure scientific excellence and mission relevance. Peer evaluation is used extensively in these merit review processes.
Nearly all research conducted by R&D performers outside the Department and its laboratory system is governed by formal processes of prospective merit review with peer evaluation and competitive selection. Such processes are codified under the Office of Energy Research's Financial Assistance Program (10 CFR Part 605), which, with some exceptions for flexibility, requires each funded grant proposal to receive a minimum of three external peer reviews. Proposals are peer reviewed for scientific excellence. This process shares many features of the merit review system of the National Science Foundation. Performance is also reviewed as part of all renewal proposals, which typically occur on three-year cycles.
Internal research programs at the Department's laboratories, likewise, undergo merit review. These reviews consist of a mix of prospective and retrospective reviews, and in many cases, both. They employ varying degrees of peer evaluation at both the laboratory and Departmental oversight levels, including regular annual reviews of program management and onsite project reviews by Departmental staff. In addition, all labs, user facilities, and major research divisions have visiting committees of outside experts that provide annual peer review of research relevance and quality.
Every internal laboratory research program is also reviewed annually by Headquarters as part of the laboratory Field Work Proposal (FWP) submission process, in accordance with the provisions of the governing M&O contracts. Field Work Proposals are the means by which the laboratories formally propose future work and seek authorization for expending R&D funds. Field Work Proposals may vary in the extent of their specificity, but in those programs that depend heavily on the use of prospective peer review in approving laboratory R&D funding, FWPs are required to be of peer review quality. Such practices are routine in the Office of Health and Environmental Research, the Experimental Plasma Research portions of the Fusion Energy Program, several major divisions of Basic Energy Sciences, and others.
In the Office of Health and Environmental Research, for example, all FWPs are required to be of peer review quality and to be externally reviewed by independent experts. Regardless of merit review method, all research projects are annually reviewed, and any project may be redirected or terminated as a result of these reviews. All new proposals are subject to merit review with peer evaluation.
Because one of the primary goals of all scientific research is to advance the forefront of knowledge, publication of original work is an essential element of the overall research activity. DOE-supported scientists, whether outside R&D performers or internal to the laboratories, are continually evaluated by the quality of their original research as published in archival, peer-reviewed journals. This publication of original work in the open literature in itself constitutes another and important form of peer review. The Department relies upon it to both guide and gauge the relevance and productivity of its internal research activities.
The Department also makes extensive use of the National Academy of Sciences and a number of standing committees constituted under the Federal Advisory Committee Act. The Office of Energy Research, for example, routinely obtains advice on program content, quality, future direction, priorities, and proposed facilities from the Basic Energy Sciences Advisory Committee, the Health and Environmental Research Advisory Committee, the High Energy Physics Advisory Panel, the Nuclear Sciences Advisory Committee, and the Fusion Energy Advisory Committee. Their expert and independent nature enable these advisory committees to provide additional and valuable outside advice used to guide the Department's R&D activities at the overall program level.
Table of
Contents
Next Section