2

Summary of Presentations

The following summaries of the presentations by James H. Turner, John C. Sommerer, J. Stephen Rottler, William F. Banholzer, Roy Levin, and Gilbert F. Decker were prepared by the workshop rapporteur, James P. McGee. Except where noted, comments in the summaries are attributable to the speakers.

Presentation by James H. Turner:
Impact of Assessments and Merit Reviews on Stakeholders

Introducing the first part of his presentation, James Turner referred the audience to a report that he had prepared for the U.S. Department of Energy (DOE) and that had been published by the Association of Public and Land-grant Universities—Best Practices in Merit Review: A Report to the U.S. Department of Energy, December 2010.1 For purposes of examining its energy portfolio, the DOE had asked how peer review should be performed at the department, and the topic was addressed at a DOE-sponsored workshop, held in January 2010, that is addressed in Best Practices in Merit Review.

Turner emphasized that assessments should include measurement against goals and intentions. Basic research and applied research are distinct. The goal of applied research is to get a task done on time and within budget. The goal of basic research is to develop science to the cutting edge and beyond.

Speakers at the January 2010 DOE workshop from industry and venture-capital research organizations showed commonality in various areas. For example, they did not want to spend time and funds evaluating proposals. Their emphasis was on their desire to secure the best team rather than worry about the details of proposals. They considered past research the best indicator of future success. They suggested that managers must be hands-on and aware that time is the enemy (the maximal time line for achieving commercialization is generally 4 years for industry and venture-capital initiatives). They suggested that if no progress is evident after a year or two, the work should be stopped and efforts directed elsewhere. They believed that every 3 to 6 months, work plans and progress should be assessed, that reviewers should be changed after 2 to 3 years, and that reviewer expertise should be based on current assessment needs. They said that the goal of the R&D efforts is to move to the marketplace. It was noted that the Department of Defense (DOD) has its “marketplace” as well, and that the DOD desires to streamline its procurement processes.

________________

1 James Turner, Best Practices in Merit Review: A Report to the U.S. Department of Energy, December 2010, Washington, D.C.: Association of Public and Land-grant Universities. Available at http://www.aplu.org/document.doc?id=2948. Accessed August 21, 2012.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 4
2 Summary of Presentations The following summaries of the presentations by James H. Turner, John C. Sommerer, J. Stephen Rottler, William F. Banholzer, Roy Levin, and Gilbert F. Decker were prepared by the workshop rapporteur, James P. McGee. Except where noted, comments in the summaries are attributable to the speakers. Presentation by James H. Turner: Impact of Assessments and Merit Reviews on Stakeholders Introducing the first part of his presentation, James Turner referred the audience to a report that he had prepared for the U.S. Department of Energy (DOE) and that had been published by the Association of Public and Land-grant Universities—Best Practices in Merit Review: A Report to the U.S. Department of Energy, December 2010.1 For purposes of examining its energy portfolio, the DOE had asked how peer review should be performed at the department, and the topic was addressed at a DOE-sponsored workshop, held in January 2010, that is addressed in Best Practices in Merit Review. Turner emphasized that assessments should include measurement against goals and intentions. Basic research and applied research are distinct. The goal of applied research is to get a task done on time and within budget. The goal of basic research is to develop science to the cutting edge and beyond. Speakers at the January 2010 DOE workshop from industry and venture-capital research organizations showed commonality in various areas. For example, they did not want to spend time and funds evaluating proposals. Their emphasis was on their desire to secure the best team rather than worry about the details of proposals. They considered past research the best indicator of future success. They suggested that managers must be hands-on and aware that time is the enemy (the maximal time line for achieving commercialization is generally 4 years for industry and venture-capital initiatives). They suggested that if no progress is evident after a year or two, the work should be stopped and efforts directed elsewhere. They believed that every 3 to 6 months, work plans and progress should be assessed, that reviewers should be changed after 2 to 3 years, and that reviewer expertise should be based on current assessment needs. They said that the goal of the R&D efforts is to move to the marketplace. It was noted that the Department of Defense (DOD) has its “marketplace” as well, and that the DOD desires to streamline its procurement processes. 1 James Turner, Best Practices in Merit Review: A Report to the U.S. Department of Energy, December 2010, Washington, D.C.: Association of Public and Land-grant Universities. Available at http://www.aplu.org/document.doc?id=2948. Accessed August 21, 2012. 4

OCR for page 4
Speakers at the January 2010 DOE workshop from organizations and agencies with a focus on basic research also shared common opinions, but their opinions differed from the opinions of those at that workshop who expressed a focus on applied research. Those who emphasized basic research considered it important to examine every proposal and proposer and to ask whether each team had the expertise, equipment, and facilities to succeed. They agreed that there is a need for competent researchers, but they also saw a need to fund new researchers and ideas. They contended that diversity of peer reviewers helps to enhance recognition of innovative proposals. Participants at the DOE workshop highlighted a need to collectively evaluate the importance both of advancing the state of the art in basic research and of performing applied research. For the second part of his presentation, Turner examined three stakeholder groups that he believes should be considered by government laboratories when assessing the satisfaction of stakeholder needs: the U.S. Congress, the Executive Branch, and the public. The Congress is a trailing indicator, rather than a leading indicator. It is difficult for Congress to influence without achieving a consensus. Congress reports to its members’ constituents, the voting public. Responding to congressional leadership is important for the members of Congress, as is being mindful of reelection. Congress is a legal society in the sense that approximately half of the U.S. Senate and a third of the members of the U.S. House of Representatives are lawyers. For the Congress, it is axiomatic that a good anecdote is the coin of the realm. So where does assessment evidence fit in? It is one step removed from the Congress. Congressional staff review assessment reports and have a key role in drafting bills. The congressional authorizing committee chair influences the Appropriations Committee chair. These are ultimate targets and prime readers of assessment reports. In general, congressional attention is thinly spread. Other targets of assessment reports are the Executive Branch, support agencies (e.g., the Congressional Research Service and the Government Accountability Office), and think tanks (e.g., the National Research Council and the Cato Institute). Reports from those institutions make a great impact. The National Research Council and other think tanks have good reputations with the Congress. The Executive Branch usually has its way during appropriations, and so influencing the Executive Branch is important. The most important stakeholder, though, is the public. Presentation by John Sommerer: Assessing R&D Organizations—Perspectives on a Venn Diagram In his opening remarks, John Sommerer noted that organizations are motivated by the desire to innovate. The context within which an R&D organization exists is 5

OCR for page 4
everything. It is not feasible to develop an assessment approach that will be applicable across all organizations—a tool kit is needed. When assessments go badly, it is because an approach is applied without regard to context. For example, academic metrics can be inappropriate for the assessment of stakeholder satisfaction. The three main influences on Sommerer’s thinking in this regard have been Vannevar Bush,2 Terence Kealey,3 and Donald Stokes.4 Vannevar Bush proposed the linear model of innovation that is now codified within the DOD (as 6.1, 6.2, etc., levels of R&D maturity). That approach, which assumes a linear transition from basic research to applied research, to development, and production, is fundamentally wrong. Rarely does innovation operate according to a linear model. Terence Kealey updated the linear model to include on-ramps, off-ramps, and feedback loops. He assumed that old science informs new science and old technology informs new technology and that only infrequently do they interact, but when they do, frequently serendipitously, significant results are produced. Kealey was hostile to the concept of government investment in R&D. This is a well-documented perspective—the view that government investment is lacking in appropriate context and motivation (it is a trailing rather than a leading indicator), and the frequent result is waste. Kealey’s perspective is that an R&D organization exists as a modus vivendi between stakeholders of the organization and its researchers, and that one cannot get people capable of embracing and understanding the cutting edge of literature to capture value for the parent organization unless one gives them money to play with. Sommerer noted that one cannot expect a staff of nonpractitioners (those who are not directly and intimately involved with the R&D activities) to cull the literature for the purpose of identifying value for an organization, because they do not have the appropriate insights. Even in a private- investment context, an R&D organization is a way of having a captive populace to extract value from cutting-edge activities. Donald Stokes identified an innovation web and emphasized the need to address very hard problems. Sommerer emphasized that an R&D organization is successful if and only if there are three components in synergistic support: alignment, vision, and people. The interaction of these components can be illustrated by a Venn diagram in which the three components are represented as overlapping sets. The alignment component is addressed by asking the following questions, and any assessment, even of technical quality, needs this context: Does the parent stakeholder have a strategy articulated with clear milestones so that it can be internalized by the organization? Does the organization have a supportive strategy? Is there a clearly articulated vision of what the parent/organization is trying to achieve according to some 2 Vannevar Bush, The Endless Frontier, Washington, D.C.: U.S. Government Printing Office, 1945. Available at http://www.nsf.gov/od/lpa/nsf50/vbush1945.htm. Accessed August 23, 2012. 3 Terence Kealey, The Economic Laws of Scientific Research, New York, N.Y., Palgrave MacMillan, 1996. 4 Donald E. Stokes, Pasteur’s Quadrant, Washington, D.C.: Brookings Institution Press, 1997. 6

OCR for page 4
milestones? Are all of these elements in synchrony? Are these strategies mutually supportive and updated? Are they good or bad strategies? Within this alignment, is the organization looking for first-mover advantages or second-mover advantages? What developments does the organization consider important to capture? The vision component is addressed by asking the following questions: Does the organization know what it wants to become (in 1-, 5-, 10-year frameworks)? What expertise is it trying to achieve? Acknowledging that strategy is about what one is going to do and not do, where does the organization choose to be a leader as opposed to being a follower of fast developments? Does the organization have expertise in areas in which it desires to be a leader, and less expertise in areas in which it desires to be a follower? Are the synergies nurtured? Are there exit strategies? Are there realistic stretch goals? Are there sufficient resources? A vision without resources is a hallucination. The component of people is addressed by the following considerations: Human capital is fundamental. Innovation requires free energy—that is, giving researchers some latitude and discretion in their work. There is no hope for the future of an organization without free energy. Peer reviews, which measure competence, have been well defined, but it is more difficult to measure motivation and external engagement. There is a need for external engagement globally in order to innovate. An assessment of human capital should ask: Are the people in the organization trying to become better? The intersection of people with alignment is addressed by the following questions: Do the people know the strategy of the organization and its parent? Are there mechanisms by which the people can contribute to the strategy? Can they interact with the organization’s customers? Are the leaders administrators or role models? What are their credentials and qualifications? Do they have a strategy to support the people of the organization? Does the organization assess and mentor the people? Does the organization have the will to release people who should not be there? Does the organization have a strategy and the resources for engagement with the external world and for encouraging such engagement? Is innovation welcome, supported, protected? External engagement must be focused on the broad global community. The intersection of vision with alignment is addressed by the following questions: Is there updating of the vision in response to changing external factors? Is there a process of self-assessment? Is there a list of lessons learned, and are they really learned, not just recorded? Is the self-assessment diligent, and does it have integrity? Is the assessment updated in acknowledgment of new strategies? There is need for both bottom-up and top- down assessment. The intersection of people with vision is addressed by the following questions: Do the people within the organization know the vision? Can the people contribute to the vision? Does the R&D organization have a strategy and appropriate resources for engagement with the larger technical community, the commercial sector, and the global community? Is innovation welcomed, supported, and protected? 7

OCR for page 4
The intersection of vision, people, and alignment is addressed by an examination of the organization’s agility, flexibility, and adaptability in the face of changing pressures, budgets, and external contexts. This intersection needs to be consciously worked by staff and leadership, and it must be internalized. There are potential cautions for external reviewers: Even robust review processes can be susceptible to inappropriate coaching. Any organization being assessed by an external group has a stake in the assessment. External assessment groups must be careful not to be used. An organization’s suggestion that the assessment be restricted to a “narrow lane” without the reviewers’ understanding the context of the organization and of the assessment is a warning sign of an attempt to influence the assessment. Given the freedom to do so, external reviewers can be very helpful in identifying suggestions for addressing the context questions mentioned here. Presentation by J. Stephen Rottler: Assessing Sandia Research J. Stephen Rottler emphasized that Sandia National Laboratories has undergone a continuous evolution of assessment of quality, relevance, and impact, with quantitative assessment evolving into qualitative assessment that is informed by data. Organizations are complex systems, composed of interconnected parts. The properties of the whole organization are not necessarily perceived by looking at individual parts. Systems behave in nonlinear ways that can be difficult to predict. Assessors must probe, watch behavior, probe, watch behavior, iteratively, being mindful that their assessment impacts behaviors. Over time, there has been a need to shift from quantitative to qualitative assessment informed by data. According to Jon Gertner’s book titled The Idea Factory: Bell Labs and the Great Age of American Innovation,5 the characteristics of Bell Laboratories, identified in that book, have been deliberately nurtured at Sandia. These characteristics, which cannot be reduced to simple rules, must be applied dynamically. They are as follows:  A critical mass of talented doers and thinkers;  An environment that encourages interaction, and an open-door policy under which experts are expected to participate in the everyday mix of work and to mentor junior staff;  Multidiscipline research teams who understand that the aim of the organization is to transform knowledge into new things;  Freedom (and time) to pursue solutions thought to be essential; and  Rich knowledge exchange between the creators and the users. 5 Jon Gertner, Bell Labs and the Great Age of American Innovation, London: Penguin Press, 2012. 8

OCR for page 4
Organizations that traditionally have been stovepiped are increasingly evolving strategies and funding approaches that acknowledge the importance of multidisciplinary research organizations. Sandia performs assessment in order to continue to improve the performance of the laboratory as an organization. Sandia addresses serious and high-consequence security challenges faced by the nation. It must deliver to requirements today and in the long term, and position itself to operate and deliver according to long-term requirements. Sandia focuses on the assurance of error prevention—developing and exhibiting behaviors so that errors are less likely to occur. Peer-review and external advisory boards examine pathways to error so that the probability of success can be maximized. There are three assessment categories: (1) Self-assessments are intended to be objective but are inherently limited. All successful organizations have mature self- assessments that are objective and that promote responsive behaviors. (2) Independent assessments conducted through external peer reviews and visiting committees (external advisory boards) are used to examine quality, relevance, impact, and responsiveness to customers. (3) Benchmarking compares an organization to other organizations and is accomplished by formal assessments (by visiting other organizations) and less formal interactions as well. Self-assessment at Sandia has become increasingly more formal and disciplined. Quarterly assessments present opportunities for leaders to examine with their teams whether their expectations about quality, relevance, and impact are being met. These assessments are performed at all levels of management. Independent assessments are performed through a research advisory board that meets twice a year. The board is composed of senior individuals drawn from across academia and the public sector. The board is used in a broad sense to assess technical quality, using external measures and comparison against other organizations. The assessment examines whether Sandia is meeting the criteria of its roles as fast follower or first researcher. It also examines the health of the research environment and connections with internal and external customers. It elucidates what is either working or getting in the way in terms of innovation. The board also meets with customers of the organization and examines the impacts of prior investments. It assesses whether investments have enabled the laboratory to continue fruitful work or to initiate new work. Laboratory Directed Research and Development (LDRD) funds are an important element of Sandia. Sandia’s director is permitted to decide how the LDRD funds are allocated across projects consistent with Sandia’s mission. The National Nuclear Security Administration (NNSA) provides oversight for this program, which captures principal-investigator-generated ideas within the management context. The program includes 5 or 6 grand challenge projects; each of these larger projects has an assigned external advisory board. Historically, these larger projects have transitioned successfully to have impact within Sandia or have achieved follow-on external funding—these impacts have been achieved with the help of the external advisory boards. 9

OCR for page 4
Assessments have traditionally been performed according to a balanced scorecard that guides the selection of data to support assessment decisions. Metrics are defined to assess three areas of measurement: value to customers, outputs, and inputs. Within each area, metrics are defined to support the assessment of what the organization is doing and how it is doing it. To assess value to customers, the value and impact in terms of leadership, stewardship, and mission satisfaction are addressed by examining measures of the effectiveness of strategic partnerships with industry and technical collaborators. To assess outputs, the excellence of scientific and technical advances is addressed within the context of management excellence, which involves measuring elements of the work environment and management assurance. To assess inputs, the capabilities of staff, technology infrastructure, and facilities are addressed by examining the science, technology, and engineering strategy through measurements of parameters indicative of the portfolio and the technical planning process. The evolving assessment processes at Sandia increasingly include an examination of qualitative factors informed by the quantitative data. The following elements are assessed: clarity, completeness, and alignment of the research strategy; alignment of the research with the organization’s missions; quality and innovation of the research; vitality of the organization’s scientists and engineers; and long- and short-term impacts of the research with respect to the organization’s missions and to advancing the frontiers of science and engineering. In summary, organizations and their assessors should be clear about the purposes of the assessment and its context; carefully decide what data to collect and what the assessment framework is; and link the assessment to the organization’s concept of what makes a great organization. Management is a qualitative sport, not a quantitative sport, but management in the absence of data is vacuous. The role of a leader in an R&D organization is to clearly express expectations to coworkers, to assess constantly whether those expectations are being met, and to take action to correct cases in which the expectations are not being met. Presentation by William F. Banholzer: An Industrial Perspective William Banholzer introduced his presentation by noting that industry does not have a right to do research—research is a privilege earned by creating value. Value means that industry can commercialize something and that someone will pay for what industry has done. Therefore, for industry, three questions must be addressed: What do people want? What will people pay for? What can they afford? Industrial organizations must convert research efforts into products that people want, will pay for, and can afford. Industry always has this compass: return on investment (ROI) of research expense versus gain. By examining impact to society (i.e., will society purchase the product?) one can define benefit beyond such metrics as scientific publications. 10

OCR for page 4
Also, there are multiple stakeholders, including stockholders, employees, government, and suppliers. Each context—industry, academia, government—has different stakeholders, whose interests must be balanced and addressed. Sometimes these different interests are not aligned. Customers are those who pay for a product; stakeholders are those who have a vested interest in an organization. For industry, customers and shareholders are those to whom the organization sells and is responsible; stakeholders may include the government, community, employees, and suppliers. For universities, customers are the students, and stakeholders are parents, the community, the faculty, and society at large. For the national laboratories, customers are the sponsoring government department and society, and stakeholders are taxpayers, the community, employees, and companies. Assessment practices must show recognition of the importance of time frame. The time frame for technology development is getting shorter and shorter. It is increasingly necessary to invest faster, and this has implications for the metrics used for assessment. The economy has also cycled—because almost all products involve chemistry, the chemical industry profits have followed the economic trends. The typical 7-year cycles are driving the need for faster research; cycles are getting shorter. Commercial launch represents enormous investment. “Materiality” refers to the point at which the cumulative investment is matched by cumulative sales. Sales do not equal profit, and so the break-even point is even farther out. Materiality points are up to 15 years out, and the cash-flow-positive point can be 25 to 30 years. In the meantime, investors are asking, “What have you done for me this quarter?” The message is that R&D is a long-term endeavor, and evaluations must consider the time frame. However, an organization cannot simply say, “Trust me for the long term.” Competitive analysis (pointing to how much competitors are spending) does not convince management. Not all technologies have same time frames; for some areas (e.g., energy) one has to think in decades. Great science does not necessarily convert into great business. Pragmatically, science is a necessary but not sufficient condition. Organizations must be wary of claims by poor science and fads. Fads have had a perturbing effect on our pragmatic research. What gets measured gets done. Measurements in industry are usually historical. R&D as a percent of sales, number of new product sales, and number of patents are woefully inadequate to define success of an industrial organization. The assumption that spending a lot yields great results is a flawed assumption. Better metrics are:  New product sales/R&D investment: a measure of productivity;  Margin on new products: new products are accountable for expanded earnings;  Patent-advantaged sales: reveals whether patents are protecting sales. 11

OCR for page 4
These metrics apply to industry, but they may not apply to national laboratories. An organization needs to define its customers and its criteria for success. Patents and publications do not indicate impact. The question to ask is, What would happen if you were not there? This question is more complicated when considering the long-term time frame. National laboratories should measure impact and some form of ROI. Successes can be communicated through anecdotes, but past history does not prove future success. It is critically important to look at an organization’s total portfolio and to examine how well the organization is spending the whole portfolio, not just the best portion of it. If an organization wants to succeed, it must play to succeed. Failures should be appreciated if they represent reasonable attempts at addressing challenges. National laboratories must consider their R&D by comparison with centers of excellence, and the national laboratories need more exposure to industry. Assessments should consider as key the question, Who are the customers and stakeholders, and how do they define success?” Presentation by Roy Levin: Managing Innovation at Microsoft Research Roy Levin emphasized that context is everything and that Microsoft Research (MSR) is in the business of innovation. MSR has laboratories in six locations around the world. Research at Microsoft is not part of the product organization. Focusing on research, MSR is funded as a corporate function. The MSR mission is to perform basic research and to advance the state of the art—which does bring advances quickly into products and services. The management challenge is to transition research into products and services. MSR contributions to Microsoft products are direct (by providing products) and indirect (by building software tools that enable product development). The role of a research organization overall is to address the long term. MSR exists to look out for the company’s future. The rationale for the existence of a research organization is to provide the capacity to deal quickly with the unknown and unexpected. Disruptive technologies, new competitors, and new business models can occur suddenly and must be responded to. An example was Microsoft’s ability, supported by its forward- looking research, to respond effectively to Bill Gates’s exhortation to attend to the evolving Internet. There are several norms in research at MSR. It conducts a broad spectrum of research in computer science and other fields, including social sciences. Research is bottom-up. Researchers, not managers, develop the projects and programs. The job of the research manager is to hire the best people and support them. Research is collaborative, both within the organization and with external collaborators such as product groups and academia. MSR maintains a flat management structure as much as possible. This affects the size of the organization—the span of control at MSR is 70 people, so the management 12

OCR for page 4
leader can interact individually with each staff member. Larger organizations can adopt this approach by breaking the research organization into smaller groups. Work at MSR is “open”—most work is published. This is key for basic research and needed for effective collaborations. A research organization cannot reap from the community without participating in it. Software is based on intellectual property. MSR seeks patent protection for defensive purposes. MSR is modeled after academia, with academic principles applied to this corporate computer science department, which is larger than that found in academic institutions. MSR seeks to publish at the right time, with emphasis on quality of publication, rather than to emphasize quantity of publications. MSR applies formal assessment mechanisms. Mechanisms include peer-reviewed publication: how work is reviewed by external peers and whether it is published in the best venues (journals and conferences). External awards and participation on program committees and editorial boards are considered as vehicles that create publications and show leadership in the field. A corporate annual review is also conducted; its mechanisms are designed to serve the larger corporation and are not focused on research. These annual reviews include feedback from peers within MSR, the rest of Microsoft, and externally, generally with attribution of the feedback comments. Formal “stack ranking” that compares researchers at similar career levels is performed, although it is sometimes tantamount to comparing trees with rocks—corporate interests generally focus on shorter terms than do those of researchers. Researchers need to have faith that the corporation will support them even though their work does not often map to the corporate evaluation cycles. The assessment process must be tweaked by the manager to consider the expected time frame for each researcher. The assessments also include formal one- on-one meetings of staff with the MSR manager. Less formal assessment mechanisms are also applied. Mentors provide feedback unencumbered by financial considerations. Mentors are internal, more senior researchers who provide career development guidance to mentees. Mentors draft the annual performance review, which is reviewed by the organization’s manager. Weekly all-hands meetings include technical talks; everyone provides a technical talk once per year. These talks are highly interactive, with discussion of innovative ideas preferred. The hiring process is also collaborative, and it is considered the most important thing done in a research organization. Across-the-lab interviewing is conducted to cover all of the relevant technical areas; it assesses the breadth and depth of a candidate’s interests and capabilities. Committees-of-the-whole involve the entire MSR organization in hiring discussions. Management-by-walking-around is a key element of informal assessment, which should not be confined to annual review. MSR evaluates what gets done, how it gets done, and when it gets done. In assessing “what gets done,” MSR looks at whether individuals and teams are advancing the field (e.g., through publication and professional service); helping Microsoft (e.g., 13

OCR for page 4
through technology transfer, consulting, or spending time embedded within a product organization); and ensuring the future (e.g., through strategic engagement with product divisions and other parts of the corporation that consider strategy). In assessing “how it gets done,” MSR evaluates collaboration (with the understanding that the best research usually involves collaboration); individual initiative; and success at mentoring others. In assessing “when it gets done,” MSR evaluates time to expected payoff and appropriate milestones. MSR recognizes that most deadlines are self-imposed but need some sense of when payoff can be expected. Ultimately management must decide whether to continue or stop a project or program. In aggregate, MSR assesses the impact of research on the field of computing and on Microsoft Corporation. Presentation by Gilbert F. Decker: Concepts for Assessment of R&D Organizations Gilbert Decker introduced his presentation by noting that industry and government, and to an extent academia, quite often use the term “research and development,” although in reality these are two somewhat different but related functions. It is constructive to discuss research and development and to show the differences in the assessment of each. The New Oxford American Dictionary defines research as both a noun and a verb. The noun is defined as “the systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions.” Then, of course, the verb is defined as “the act of conducting research.” The term “development” is best defined here as “the creation of a new or refined product or idea.” Said another way, research is usually based on a hypothesis and on the design of an experiment or analyses to confirm or disprove the hypothesis; development is the creation of a product. Having defined terms, it is useful to discuss the assessment of development programs, followed by a discussion of the assessment of research programs and then a comparison of the two. Usually the objective of a development program is to produce a product that has applications and can be sold in the market and/or used by mission organizations, such as the military, to accomplish needed functions. The beginning of a development program, to be successful, requires fairly rigorous descriptions of the functions that the product must perform and specifications that drive the design. Consequently, the assessment of the quality and success of a development program is based on some definable metrics. Further, processes of managing the development program also need to be well defined, although they are not always well executed. One very key factor in the decision to begin a development program is the maturity of the technology. If the program is not based on known facts and conclusions established from proven research results, it is clearly unwise to pursue it. If one believes in the program, but it is not ready, it is best to recycle it back into applied research. Additionally, a development program usually has fairly rigorous cost estimates to which it tries to adhere. 14

OCR for page 4
So, the assessment of a development program has fairly straightforward metrics, based primarily on two parameters: how well the program functions compared to its intended functions and specifications, and how well it adheres to its cost estimates. Both of these assessment criteria can be measured by means of a well-designed test program. Research is often described in two categories: basic research and applied research. A descriptive definition of “basic research” is that which is carried out to increase understanding of fundamental principles. It is not intended to yield immediate commercial benefits; basic research can be thought of as arising out of curiosity. However, in the long term, it is the basis for many commercial products and for applied research. Thus, basic research is a quest for knowledge and understanding. A descriptive definition of “applied research” is that which is designed to solve actual problems, rather than to acquire new knowledge or theory for its own sake. The objective of applied research is to define new concepts that are based on knowledge, theory, and understanding from basic research. There is no sharp dividing line between basic and applied research. Both categories are often carried out in research organizations, such as at research universities and in government laboratories. Often, the results of basic research become evident, and the research team may morph into an applied research program. There are two dimensions in assessing research and development activities: quality and relevance. Assessing quality is the focus of evaluating the technical merit of science and engineering work. The following issues are germane to the consideration of both quality and relevance: the adequacy of the resources available to support high- quality work; the effectiveness of the organization’s delivery of the services and products required to fulfill its goals and mission and to address the needs of its customers; the degree to which the organization’s current and planned R&D portfolio supports its mission; and the elements of technical management that affect the quality of the work. As with development programs, metrics are required to assess the quality and relevance of research programs. Such metrics for research programs are not uniformly numerical measurements; or if numerical scoring is desired, the numerical assessment may require that judgments be made by skilled and experienced people, including peers and research managers. In the case of basic research, relevance should not be weighted as heavily as quality. That is simply because of the definition and goal of basic research: the quest for new knowledge and understanding for its own sake. One should focus on quality of the research organization itself and also on the quality of the individual research programs. Assessments of applied research should address both relevance and quality. The candidate metrics for assessing the quality and relevance of basic and applied research, presented below, are drawn in large part from two studies: Managing Air Force 15

OCR for page 4
Basic Research6 and Improving Army Basic Research.7 Gilbert Decker was a member of the committee that conducted the former study and chaired the committee that conducted the latter study. Metrics for assessing the technical merit and quality of the science and engineering work include the following:  Membership in professional societies. A high percentage (perhaps at least 75 percent) of the researchers of the organization should be members of professional organizations.  Memberships in the National Academies and/or high recognition by professional societies. A very low percentage would be indicative of overall inadequacy of staff.  Number of members of research staff who have been awarded National Medals of Science. There should certainly be at least two or three who have received this award. This is probably a good indicator of the overall quality of the research staff, the theory being that winners of these medals would not be part of a low-quality research organization.  Whether the research organization maintains a database of research projects, findings and results. This database should also include lists of publications in refereed journals, citations, and awards. It should also include, based on the findings and results, an assessment of lessons learned by the researchers on each project. Lessons learned from failed projects can be as valuable as those learned from successful projects. Such a database could enable an assessment by funding organizations and scientific peer-review committees of the quality of basic research as well as applied research.  Percentage of research staff members with doctorates and/or postdoctorate fellowships. Something greater than 50 percent is the metric for recognized high-quality research organizations.  The balance between internally sponsored basic and applied research funding and customer funding that seeks applied research and/or engineering support to address specific problems. High-quality research organizations have 10 to 15 percent of their total funding in internally generated basic and applied research projects; another 25 to 30 percent is devoted to applied research funded by external organizations looking for concept solutions to problems; the remainder is allocated to scientific and engineering support to advanced development programs. When the internally generated basic and applied research effort falls significantly below 10 percent, the overall quality and stature of the research organization diminish significantly. The following are management functions: providing the resources available to support high-quality work, effectively delivering the services and products required to 6 National Research Council, Managing Air Force Basic Research, Washington, D.C.: National Academy Press, 1993. 7 G. Decker, J. Davis, R.A. Beaudet, S. Dalal, and W.H. Forster, Improving Army Basic Research: Report of an Expert Panel on the Future of Army Laboratories, Santa Monica, Calif.: RAND Arroyo Center, 2012. 16

OCR for page 4
fulfill the organization’s goals and mission and to address the needs of its customers, and maintaining a current and planned R&D portfolio that supports the organization’s mission. Generally, managers or directors of research organizations should be scientifically trained personnel with research experience. There are certainly exceptions, but they are few; directors must genuinely understand and believe in basic and applied research. The manager must be capable of assessing and prioritizing research proposals from the research staff and from external funding sources and must also ensure that the aforementioned database is rigorously maintained. The manager must also be the marketing leader in finding research grants for internally generated basic and applied research proposals; must support customers who are seeking solutions to problems using applied research; and must diligently push for balance between internally generated basic and applied research proposals, customer-funded basic research proposals, and customer- funded advanced engineering support. Non-academic organizations (e.g., corporations and government organizations) which include R&D organizations that are productive in both research and engineering are most successful when the R&D director or manager reports to the overall leader of the organization. The following ideas for assessment processes might be considered:  The research organization should conduct internal assessments of about 25 percent of basic and applied research projects, randomly selected each year, using the assessment criteria.  The research organization should have a “blue ribbon” external panel of scientific experts to assess the internally reviewed projects with respect to the assessment criteria. The external review should be compared with internal assessments, using the same criteria.  The research organization should solicit feedback from external customers who fund projects. The solicitation for feedback should be oriented around customer satisfaction. The feedback survey should be a tailored version of the assessment criteria.  The research organization should review the research staff composition annually with regard to the quality of staff and mix of staff expertise. 17