Click for next page ( 219


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 218
Appendix F Science, Technology, and Innovation Databases and Heat Map Analysis Leland Wilkinson and Esha Sinha 1 The panel assembled and analyzed underlying data on research and development (R&D), science and technology (S&T), human capital, and innovation to determine the following: What are the primary indicators that are necessary for the National Center for Science and Engineering Statistics (NCSES) to disseminate, and are they produced by traditional or frontier methods? To address this question, cluster analysis, primarily a heat map tool, was used together with knowledge gleaned from the literature on the performance of science, technology, and innovation (STI) indicators. Reference is made to the National Science Board’s Science and Engineering Indicators (SEI) biennial publication when appropriate, but this analysis is not a full review of the SEI publication. What are the redundant indicators that NCSES does not need to produce going forward? These indicators might be low performers; highly correlated with other, more useful indicators; or measures that are gathered by other organizations. NCSES could target these indicators for efficiency gains while curating the statistics that are in demand but reliably produced elsewhere. This appendix describes the main data on R&D, S&T, human capital, and innovation that the panel assembled and analyzed. It is divided into four sections. The first two sections contain descriptions of data sources from NCSES and other international statistical organizations. The third section presents the heat map analysis, citing the literature on methodological underpinnings of this technique. The final section gives observations based on this analysis. Not all of the data sources described were analyzed, as it was not feasible to investigate such a wide variety of data culled from various sources. Only databases of the five main STI data providers were analyzed: NCSES, OECD; Eurostat; the United Nations Educational, Scientific and 1 Esha Sinha, CNSTAT staff, compiled the data used in the heat map analysis. Leland Wilkinson, panel member, initially ran the heat map program, based on an algorithm that he developed. Dr. Sinha then ran several versions of the program on different datasets and over several different time periods. She presented the results of the heat map analysis to the panel during its April 2012 panel meeting. She subsequently ran more sensitivity analyses to ensure the stability of the results. Panel member John Rolph reviewed the work, concluding that the statistical analysis was sound and potentially instructive as an indicators prioritization exercise that NCSES might perform in the future. PREPUBLICATION COPY: UNCORRECTED PROOFS F-1

OCR for page 218
F-2 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY Cultural Organization (UNESCO), Institute of Statistics (UIS); and Statistics Canada. Indicators published in the SEI 2012 Digest were also analyzed. ASSEMBLED DATA National Center for Science and Engineering Statistics NCSES communicates its S&T data through various publications, ranging from InfoBrief’s to Detailed Statistical Tables (DSTs) derived using table generation tools. The three table generation tools—the Integrated Science and Engineering Resource Data System (WebCASPAR), the Scientists and Engineers Statistical Data System (SESTAT), and the Survey of Earned Doctorates (SED) Tabulation Engine (National Center for Science and Engineering Statistics, 2013b)—are each supported by application-specific database systems. The Industrial Research and Development Information System (IRIS) is an additional searchable database of prepopulated tables. WebCASPAR hosts statistical data for science and engineering (S&E) at U.S. academic institutions (National Science Foundation, 2012e). This database is compiled from several surveys, including: — SED 2/Doctorate Records File; — Survey of Federal Funds for Research and Development; — Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions; — Survey of Research and Development Expenditures at Universities and Colleges/Higher Education Research and Development Survey; — Survey of Science and Engineering Research Facilities; — National Science Foundation (NSF)/National Institutes of Health (NIH) Survey of Graduate Students and Postdoctorates in Science and Engineering; and — National Center for Education Statistics (NCES) data sources—Integrated Postsecondary Education Data System (IPEDS): - IPEDS Completions Survey; - IPEDS Enrollment Surveys; - IPEDS Institutional Characteristics Survey (tuition data); and - IPEDS Salaries, Tenure, and Fringe Benefits Survey. SESTAT (National Science Foundation, 2013d) is a database of more than 100,000 scientists and engineers in the United States with at least a bachelor’s degree. This is a comprehensive data collection on education, employment, work activities, and demographic characteristics, covering 1993 to 2008. 3 The SESTAT database includes data from: 2 SED data on race, ethnicity, citizenship, and gender for 2006 and beyond are available in the SED Tabulation Engine. All other SED variables are available in WebCASPAR except for baccalaureate institution. For more details on the WebCASPAR database, see https://webcaspar.nsf.gov/Help/dataMapHelpDisplay.jsp?subHeader=DataSourceBySubject&type=DS&abbr=DRF &noHeader=1. 3 Data for 2010 are forthcoming in 2013. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-3 — the National Survey of College Graduates (NSCG); — the National Survey of Recent College Graduates (NSRCG); — the Survey of Doctorate Recipients (SDR); and — an integrated data file (SESTAT). IRIS (National Center for Science and Engineering Statistics, 2013a) is a database containing industrial R&D data published by NSF from 1953 through 2007. It comprises more than 2,500 statistical tables, which are constructed from the Survey of Industrial Research and Development (SIRD). It is, therefore, a databank of statistical tables rather than a database of microdata of firm-specific information. The data are classified by Standard Industrial Classification and North American Industrial Classification codes (as appropriate), and by firm size, character of work (basic, applied, development), and state. Employment and sales data for companies performing R&D are also included in IRIS. The data outlined above focus on academic and industrial R&D expenditures and funding and on human capital in S&T. NCSES conducts five surveys to capture R&D support and performance figures for various sectors of the economy. The National Patterns of Research and Development Resources series of publications presents a national perspective on the country’s R&D investment. R&D expenditure and performance data are available, as well as employment data on scientists and engineers. The National Patterns data are useful for international comparisons of R&D activities, and they also report total U.S. R&D expenditures by state. The data series span 1953 through 2011, and are a derived product of NCSES’s above-referenced family of five active R&D expenditure and funding surveys: Business Research and Development and Innovation Survey (BRDIS; for 2007 and earlier years, the industrial R&D data were collected by the SIRD); Higher Education Research and Development Survey (HERD; for 2009 and earlier years, academic R&D data were collected by the Survey of Research and Development Expenditures at Universities and Colleges); Survey of Federal Funds for Research and Development; Survey of Research and Development Expenditures at Federally Funded R&D Centers (FFRDCs); and Survey of State Government Research and Development. 4 4 For details on each of these surveys, see http://nsf.gov/statistics/question.cfm#ResearchandDevelopmentFundingandExpenditures [November 2012]. A sixth survey, the Survey of Research and Development Funding and Performance by Nonprofit Organizations, was conducted in 1973 and for the years 1996 and 1997 combined. The final response rate for the 1996-1997 survey was 41 percent (see http://www.nsf.gov/statistics/nsf02303/sectc.htm). This lower-than-expected response rate limited the analytical possibilities for the data, and NSF did not publish state-level estimates. The nonprofit data cited in National Patterns reports either are taken from the Survey of Federal Funds for Research and Development or are estimates derived from the data collected in the 1996-1997 survey. See National Science Foundation (2013c, p. 2), which states: “Figures for R&D performed by other nonprofit organizations with funding from within the nonprofit sector and business sources are estimated, based on parameters from the Survey of R&D Funding and Performance by Nonprofit Organizations, 1996-97.” PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-4 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY The SEI biennial volume is another notable contribution from NCSES, published by the National Science Board. It not only contains tables derived from the table generation tools described above but also amalgamates information from NCES surveys, administrative records such as patent data from government patent offices, bibliometric data on publications in S&E journals, and immigration data from immigration services. For example, tables on the U.S. S&E labor force are generated using data from the American Community Survey, the Current Population Survey (U.S. Census Bureau), SESTAT, and Occupational Employment Statistics (Bureau of Labor Statistics) (National Science Board, 2012a, Table 3-1, p. 3-8). Along with information on U.S. R&D capacity and outputs, the SEI Digest 2012 contains analysis of the data. The SEI indicators can be classified as follows (National Science Board, 2012b) (see Box F-1): (1) global R&D and innovation; (2) U.S. R&D funding and performance; (3) U.S. R&D federal portfolio; (4) science, technology, engineering, and mathematics (STEM) education; (5) U.S. S&E workforce trends and composition; (6) knowledge outputs; (7) geography of S&T; and (8) country characteristics. BOX F-1 NCSES’s STI Indicators 1. Global research and development (R&D) and innovation Worldwide R&D expenditure by regions and countries Average annual growth of R&D expenditure for the U.S., European Union (EU), and Asia-10 economies Annual R&D expenditure as share of economic output (R&D/gross domestic product [GDP]) R&D testing by affiliation, region/country U.S. companies reporting innovation activities Exports and imports of high-tech goods 2. U.S. R&D funding and performance (including multinationals and affiliates) U.S. R&D expenditure by source of funds (including venture capital) Types of U.S. R&D performed Types of U.S. R&D performed by source of funds U.S. academic R&D expenditure by source of funds 3. U.S. R&D federal portfolio U.S. federal R&D expenditure by type of R&D U.S. federal support for science and engineering (S&E) fields U.S. federal R&D budget by national objectives U.S. federal R&D spending on R&D by performer Federal research and experimentation tax credit claims by North American Industrial Classification System (NAICS) industry Federal technology transfer activity indicators Small Business Innovation Research (SBIR) and Technology Innovation Program 4. Science, technology, engineering, and mathematics (STEM) education (most measures have demographic breakouts) Average mathematics and science scores of U.S. students (National Assessment of Educational Progress [NAEP] and Programme for International Student Assessment [PISA]) Teacher participation, degrees, and professional development High school students taking college classes First university degrees in natural sciences and S&E fields by country/region PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-5 S&E degrees, enrollments, and related expenditures—associate’s, bachelor’s, master’s, doctoral Doctoral degrees in natural sciences and S&E fields by country/region Distance education classes 5. U.S. S&E workforce trends and composition Individuals in S&E occupations and as a percentage of the U.S. workforce S&E work-related training Unemployment rate for those in U.S. S&E occupations Change in employment from previous year for those in STEM and non-STEM U.S. jobs Women and underrepresented minorities in U.S. S&E occupations Foreign-born percentage of S&E degree holders in the United States by field and level of S&E degree 6. Knowledge outputs S&E journal articles by region/country Engineering journal articles as a share of total S&E journals by region/country Citations in the Asian research literature to U.S., EU, and Asian research articles Patents and citations of S&E articles in United States Patent and Trademark Office (USPTO) patents U.S. patents granted to non-U.S. inventors by region/country/economy Share of U.S. utility patents awarded to non-U.S. owners that cite S&E literature Value added of knowledge and technology 7. Geography of S&T Location of estimated worldwide R&D expenditure Average annual growth rates in number of researchers by country/economy Value added of high-tech manufacturing by region/country Exports of high-tech manufactured goods by region/country Cross-border flow of R&D funds among affiliates State S&T indicators 8. Country characteristics Macroeconomic variables Public attitudes toward and understanding of S&T SOURCE: National Science Board, 2012b. The primary conclusions of the SEI Digest are drawn from the indicators outlined above and are supported by more detailed STI data collected by NCSES. The list of variables is presented in Table F-1. Because of space limitations, it was not possible to highlight in this table the fact that most of the information in the SEI—such as assessment scores, S&E degrees, individuals in S&E occupations, R&D expenditures and their various components, federal R&D obligations, patents, and venture capital—is available at the state level. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-6 TABLE F-1 Subtopics of Science, Technology, and Innovation Data Produced by Agencies/Organizations, Showing Level of Detail and Unique Variables (Highlighted Text) Agency/ Organization Total R&D Industrial R&D Academic R&D Federal R&D GBAORD Nonprofit R&D NCSES Total R&D by Industrial R&D by Academic R&D Federal R&D by R&D Nonprofit R&D (NB: Statistics on performer and funder, character of by funder, funder, character obligations and by funder, R&D expenditure and funder, character work, NAICS character of of work outlays by character of SEH degrees of work classification, work; entities character of work, S&E available by state company size and work and field, subrecipients of performing extramural academic R&D sector; reported entity, type of in Science and nonprofit Engineering organization Indicators only NCSES and NSB Total R&D by Industrial R&D by Academic R&D Federal R&D by Federal Domestic R&D performer and funder, NAICS in S&E and major obligations for performed by funder, character classification, non-S&E fields socioeconomic R&D and R&D private of work, country/ company size; R&D objectives, plant by agency, nonprofit economy performed by country/region performer, sector, domestic multinational character of R&D funded by companies, foreign work private affiliates nonprofit sector Statistics Canada GERD by BERD by funding HERD GOVERD by Private performer and sector socioeconomic nonprofit R&D funder objectives, type by funder of science, components OECD GERD by BERD by funding HERD, HERD GOVERD, GBAORD by GERD performer, sector, type of cost, financed by GOVERD socioeconomic performed by funder, field of size class, field of industry financed by objectives private science, science, performing industry nonprofit sector socioeconomic industry objectives PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-7 Eurostat GERD by BERD by funding HE intramural Government GBAORD by GERD (NB: Almost all data funding source, source, type of cost, expenditure by intramural socioeconomic performed by available at regional sector of size class, economic funding source, expenditure by objectives private level) performance, activity type of cost, funding source, nonprofit type of cost, field of science, sector of sector; GERD socioeconomic socioeconomic performance, funded by objectives, field objectives type of cost, private of science socioeconomic nonprofit sector objectives, field of science UNESCO GERD by GERD performed GERD GERD performing by business performed by performed by sector and enterprise sector; higher private funding sector, GERD funded by education nonprofit field of science, business enterprise sector; GERD sector; GERD character of work sector funded by funded by higher private education sector nonprofit sector Technology BOP and Science, Public International Engineering, and Attitudes Trade in R&D- R&D Personnel, Health Degrees; toward Agency/ Intensive Patents and Venture Scientists and Assessment Science and Organization Industries Capital Engineers Scores Innovation Technology NCSES Scientists and Graduate students, Product and (NB: engineers by doctorate holders, process Statistics on gender, age, postdoctorates, innovation by R&D race/ethnicity, nonfaculty NAICS expenditure, level of highest research staff by classification SEH degrees degree, gender, available by occupation, labor race/ethnicity, state) force status, citizenship, employment academic field, sector, Carnegie primary/secondary classification; work activity, SEH doctorates by PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-8 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY median annual gender, age, salaries race/ethnicity, occupation, labor force status, employment sector, primary/secondary work activity, postdoctoral appointments, median annual salaries NCSES and U.S. trade balance USPTO patents granted Workers in S&E SEH doctorate Small Business Media NSB in research, by selected technology and STEM holders by gender, Innovation coverage, news development, and area, region/country/ occupations by race/ethnicity, Research stories by topic testing services by economy; patenting MSA, occupation field of doctorate, funding per area; correct affiliation; exports activity in clean energy category, sector of $1 million of answers to S&T of high- and pollution control educational employment, GDP by state and S&T- technology and technologies; patents background, R&D academic related manufactured granted by BRIC nations work activities, appointment, questions by products by by share of resident and age, salary, gender and technology level, nonresident inventors; ethnicity/race; unemployment country/region; product, region/ patent citations to S&E scientists and rate; S&E public country/economy; articles by patent engineers doctorate perceptions of global value technology area, article reporting recipients and various added by type of field; patenting activity international full-time S&E occupational industry; ICT of employed U.S.-trained engagement by graduate students groups’ infrastructure SEH doctorate holders; demographic by source, contribution to index; U.S. high- U.S. venture capital characteristics, primary society and technology investment by financing education, mechanism of public policy- microbusinesses stage and employment support, Carnegie making industry/technology; sector, classification; process; public venture capital disbursed occupation, foreign recipients assessment/ per $1,000 of GDP; salary, work- of U.S. S&E opinion of stem venture capital deals as a related training; doctorates by cell research percentage of high- foreign-born field, and PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-9 technology business workers in S&E country/economy environmental establishments; venture occupations by of origin; field problems; capital disbursed per education level switching among source of venture capital deal by postsecondary information for state students; time S&T issues taken to receive an S&E doctorate; community college attendance among recent recipients of S&E degrees by sex, race/ethnicity, degree level, degree year, citizenship status; NAEP assessment scores in mathematics and science; advanced placement exam taking by public school students Statistics Intellectual property Researchers, University Innovation Canada commercialization by support staff, degrees, diplomas, activities; higher education sector; technicians by and certificates product and intellectual property R&D performing granted by process management by federal sector and type of program level and innovation; department and agency science; federal Classification of degree of personnel engaged Instructional novelty; in S&T by Programs, gender, hampering activity, type of immigration status factors of and science, S&T obstacles to component innovation; important sources of PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-10 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY information; cooperation arrangements; innovation impacts; methods of protection; geomatic activities OECD BOP—payments Patent applications, R&D personnel Graduates by field and receipts; Triadic Patent Families, by field of and level of trade—imports patents in selected science, sector of education; PISA and exports by technologies by region, performance, scores in science R&D-intensive international cooperation qualification; and mathematics industries researchers by sector of performance and gender Eurostat Trade in high-tech Patent applications at Human resources Doctorate holders Innovation (NB: Almost industries and USPTO and EPO by in S&T, R&D by gender, activity activities; all data knowledge- priority year and sector, personnel, and status; employed product and available at intensive sectors ownership of patents, researchers by doctorate holders process regional within EU and patent citations; gender, field of by gender, sector, innovation; level) ROW; European and science, sector of occupation, field degree of employment in international performance, of science, job novelty; these sectors by copatenting, Triadic qualification; mobility hampering gender, Patent Families citizenship of factors of and occupation, researchers in public funding educational government and for innovation; qualification, higher education important mean earnings sector sources of information; cooperation arrangements; environmental PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-11 innovation; objectives of innovation; impacts of innovation; methods of protection; employees in innovation sector UNESCO R&D personnel Innovation in and researchers manufacturing (FTE and head sector—firms count) by gender, involved in performing sector, innovation; educational cooperation qualification, field arrangements; of science; hampering technicians and factors of other supporting innovation; staff by available for 12 performing sector nations NOTE: BERD = business enterprise expenditure on research and development; BOP = balance of payments; BRIC = Brazil, Russia, India, and China; EPO = European Patent Office; EU = European Union; FTE = full-time equivalent; GBAORD = government budget appropriations or outlays for research and development; GDP = gross domestic product; GERD = gross domestic expenditure on research and development; GOVERD = government intramural expenditure on research and development; HE = higher education; HERD = higher education expenditure on research and development; ICT = information and communication technology; MSA = metropolitan statistical area; NAEP = National Assessment of Educational Progress; NAICS = North American Industry Classification System; NSB = National Science Board; NCSES = National Center for Science and Engineering Statistics; PISA = Programme for International Student Assessment; R&D = research and development; ROW = rest of the world; S&E = science and engineering; SEH = science, engineering, and health; STEM = science, technology, engineering, and mathematics; UNESCO = United Nations Educational, Scientific and Cultural Organization; USPTO = United States Patent and Trademark Office. SOURCES: Adapted from BRDIS, see: http://www.nsf.gov/statistics/industry/ [November 2012]. Federal Funds, see: http://www.nsf.gov/statistics/fedfunds/ [November 2012]. R&D Expenditure at FFRDCs, see :http://www.nsf.gov/statistics/ffrdc/ PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-48 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY SOURCES: Adapted from National Science Foundation (2010). NSF Releases New Statistics on Business Innovation. NSF 11-300. National Center for Science and Engineering Statistics, available: http://www.nsf.gov/statistics/infbrief/nsf11300/[November 2012]. Statistics Canada, Adapted from CANSIM Table 358-00321, 2 Survey of Innovation, selected service industries, percentage of innovative business units [November 2012]. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-49 TABLE F-7 Innovation Statistics from NCSES and Eurostat PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-50 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY SOURCES: Adapted from National Science Foundation. (2010). NSF Releases New Statistics on Business Innovation. NSF 11-300. National Center for Science and Engineering Statistics, available: http://www.nsf.gov/statistics/infbrief/nsf11300/[November 2012]. Eurostat, see http://epp.eurostat.ec.europa.eu/portal/page/portal/science_technology_innovation/data/database European Union, 1995- 2013[November 2012]. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-51 TABLE F-8 Innovation Data from U.S. Agencies Other Than NCSES Database/Survey/Data Agency Collection Mechanism Indicator/Data Items Time Period U.S. Patent and Patent Assignments Patent assignments and 2010 onward Trademark Office Dataset change of ownership of (USPTO) patents that are granted by USPTO Trademark Casefile Trademarks granted by 1884-2010 Dataset USPTO Trademark Assignments Change of ownership of 2010 onward Dataset trademarks granted by USPTO Economic Research Rural Establishment Inventory of innovation First survey cycle will Service, U.S. Innovation Survey activities; use of be conducted in 2013 Department of technology by labor Agriculture force, establishment, and community characteristics; factors hampering innovation; funding source for innovation; applications for intellectual property and trademarks; sources of information on new opportunities National Institutes of STAR METRICS Recipient-based data 2009-2012 Health, National containing information Science Foundation, and on contract, grant, and White House Office of loan awards made under Science and Technology the American Recovery Policy and Reinvestment Act of 2009 National Science Research Spending and Recipient-based data 2007 onward Foundation Results containing information on awards made by the National Science Foundation and the National Aeronautics and Space Administration National Institute of Federal Laboratory Patent applications, 1987-2009 Standards and (Interagency) invention licenses, Technology, Technology Transfer cooperative R&D Department of Summary Reports agreements, R&D Commerce obligations—extramural and intramural PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-52 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY Small Business Small Business Federal R&D funds/ 1983-2012 Administration Innovation Research awards by agency, (SBIR) and Small category, and state Business Technology Transfer (STTR) programs Department of Energy Energy Innovation Issued U.S. patents and 1979 onward Portal—Visual Patent published patent NB: Information Search Tool applications that are available at the state, created using county, and municipal Department of Energy levels, as well as from funding utilities and nonprofits Advanced Energy-saving Manufacturing Office— incentives and resources State Incentives and available for Resource Database commercial and industrial plant managers SOURCES: USPTO Databases, see: http://www.gwu.edu/~gwipp/Stuart%20Graham%20020712.pdf. Rural Development, USDA, see: http://www.gpo.gov/fdsys/pkg/FR-2011-06-22/html/2011-15474.htm. STAR METRICS, see: https://www.starmetrics.nih.gov/Star/Participate#about. Research Spending and Results, see: https://www.research.gov/research- portal/appmanager/base/desktop?_nfpb=true&_eventName=viewQuickSearchFormEvent_so_rsr Federal Laboratory (Interagency) Technology Transfer Summary Reports, see: http://www.nist.gov/tpo/publications/federal-laboratory-techtransfer-reports.cfm. SBIR and STTR, see: http://www.sbir.gov/. Energy Innovation Portal, see: http://techportal.eere.energy.gov/. Advanced Manufacturing Office, see: http://www1.eere.energy.gov/manufacturing/. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-53 TABLE F-9 Innovation Product Data from Private Sources Database/Survey/Data Agency Collection Mechanism Indicator/Data Items Time Period University of California, Connecting Outcome Integrates data on 2007-2012 Los Angeles Measures in government grants, Entrepreneurship, dissertations, patents, Technology, and and publicly available Science (COMETS) firm data; currently database contains information on patents granted by the U.S. Patent and Trademark Office (USPTO) and on National Science Foundation (NSF) and National Institutes of Health (NIH) grants Association of University Statistics Access for Academic licensing 1991-2010 Technology Managers Tech Transfer data from participating (STATT) academic institutions: licensing activity and income, start-ups, funding, staff size, legal fees, patent applications filed, royalties earned Association of Public and New Metrics to Relationship with Pilot conducted in Land-grant Universities— Measure Economic industry: agreements, spring 2012 with 35 Commission on Impact of Universities clinical trials, participating Innovation, sponsored research, institutions Competitiveness, and external clients Economic Prosperity Developing the regional and national workforce: student employment, student economic engagement, student entrepreneurship, alumni in workforce Knowledge incubation and acceleration programs: success in knowledge incubation and acceleration programs, ability to attract investments, relationships between clients/program participants and host PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-54 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY university Harvard University Patent Network Patent coauthorship 1975 onward Dataverse: U.S. Patent network Inventor Database PricewaterhouseCoopers Money Tree Report Venture capital firms Quarterly data, 1st and National Venture and firms receiving quarter 1995 onward Capital Association financing: quarterly and yearly investment amounts, number of deals by industry, stage of development, first- time financings, clean technology, and Internet-specific financings CB Insights, Venture Deal, Venture Capital Profile of venture Grow Think Research, Database capital firms and Dow Jones VentureSource venture capital- financed firms SOURCES: COMETS Database, see: http://scienceofsciencepolicy.net/?q=node/3265. STATT Database, see: http://www.autm.net/source/STATT/index.cfm?section=STATT. APLU Economic Impact, see: http://www.aplu.org/page.aspx?pid=2693. Patent Network Dataverse, see: http://thedata.harvard.edu/dvn/dv/patent. Money Tree Report, see: https://www.pwcmoneytree.com/MTPublic/ns/index.jsp. Venture Capital Databases, see: http://www.cbinsights.com/; http://www.venturedeal.com/; http://www.growthinkresearch.com/; https://www.venturesource.com/login/index.cfm?CFID=2959139&CFTOKEN=53e4cab- 1e600d5d-9089-411f-a010-949554ae0978. OBSERVATIONS Many Nations Analysis Figures F-4 to F-6 show a cluster heat map, a hierarchical cluster tree, and the multidimensional scaling of a Pearson correlation matrix, respectively. The input matrix consists of the main S&T variables from the OECD, UNESCO, and Eurostat databases. The red and orange squares along the diagonal of the heat map in Figure F-4 show that those variables are very closely related to each other, and either they could be merged, or the most well-behaved and consistent variables among them could be selected. Figures F-5 and F-8 show clusters of variables. Broadly speaking, human resource variables form one category and trade variables another. Figures F-6 and F-9 show sets of variables that are either similar or dissimilar. In these two figures, the dimensions have no interpretation, and one is looking for clusters of variables that would indicate they belong together. Strong correlation patterns are observed in the variables on researchers, technicians, and other supporting stuff. These variables are closely grouped together. Moreover, within these variables, those produced by the same organization are more similar. Clusters of subtopics are also observed. Expenditure variables, trade variables, and patent variables are more similar to variables within their group. This shows that variables on a PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-55 subtopic relay similar information; i.e., they are proxy variables. For example, if an analyst is looking at predictor variables for a regression model and is unable to obtain data on technical staff, researchers can substitute. In some ways, this relieves the burden on statistical agencies/offices trying to follow the Frascati Manual’s recommendations. Even if they fall short in collecting certain variables, similar information can be gleaned from other variables on the same topic. Single indicators highlighted for each subtopic as primary indicators are not shown here, as that would lead to conjecture. Nations should decide which variables to collect depending on ease of collection and budgetary constraints. The panel is not asserting that statistical offices around the world should stop collecting detailed S&T data, as the utility of variables is not limited to the ability to feed them into a regression model. National statistical offices collect detailed STI information through surveys and/or by using administrative records to answer specific policy questions, such as the mobility of highly skilled labor, the gender wage gap in S&T occupations, and the amount of investment moving into certain S&T fields. In can be said that the S&T community is interested in understanding the progress of nations in attracting the best talent, or the broad careers pursued by Ph.D. holders in particular fields, or the R&D investment in environmental projects. The main concern faced by the panel was the unavailability of detailed data as main variables undergo disaggregation. Apart from OECD and Eurostat member countries, the rest of the world has yet to keep pace in terms of capturing STI information in accordance with recommendations of the Frascati and Oslo Manuals. OECD and Eurostat have been frontrunners in pursuing valuable information, and they should be commended for their efforts. At the same time, the panel is not critical of non-OECD and non- Eurostat nations, as both data collection agencies and respondents must undergo a learning process to provide such fine data in a consistent fashion. Figures F-14 to F-16 show a cluster heat map, a hierarchical cluster tree, and the multidimensional scaling of a Pearson correlation matrix, respectively. The input matrix consists of S&E indicators from the SEI 2012 Digest. The red and orange squares along the diagonal of the heat map show that those variables are very closely related to each other, and either they could be merged, or the most well-behaved and consistent variables among them could be selected. Figure F-15 shows clusters of indicators; Figure F-16 shows sets of indicators that are either similar or dissimilar. In these two figures, the dimensions have no interpretation, and one is looking for clusters of variables that would indicate they belong together. Indicators representing the service sector are observed to be highly correlated with each other. Indicators denoting first university degrees are closely grouped together. The same conclusion can be drawn for indicators on generation of S&E knowledge (articles and citations). Therefore, clusters of subtopics are observed, similar to those observed for STI variables from the OECD, Eurostat, and UNESCO databases. Certain indicators, such as R&D as a share of GDP, global high-value patents, doctoral degrees in engineering, and doctoral degrees in natural science are not strongly correlated with other indicators. Hence within the set of indicators analyzed, these four indicators stand apart. The reader should not assume that these indicators are unique, as the list of indicators analyzed here is small. The uniqueness might not hold if more indicators were included in the input matrix. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-56 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY FIGURE F-14 Heat map of science and engineering indicators from SEI 2012 Digest. NOTE: GDP = gross domestic product; KIS = knowledge-intensive services; RD = research and development; S&E = science and engineering; SS = social services; VA = global value added. SOURCE: Panel Analysis and S&E I 2012, see http://www.nsf.gov/statistics/seind12/tables.htm [November 2012]. FIGURE F-15 Hierarchical cluster of science and engineering indicators from SEI 2012 Digest. NOTE: GDP = gross domestic product; KIS = knowledge-intensive services; RD = research and development; S&E = science and engineering; SS = social services; VA = global value added. SOURCE: Panel Analysis and S&E I 2012, see http://www.nsf.gov/statistics/seind12/tables.htm [November 2012]. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
SCIENCE, TECHNOLOGY, AND INNOVATION DATABASES AND HEAT MAP ANALYSIS F-57 FIGURE F-16 Multidimensional scaling of science and engineering indicators from SEI 2012 Digest. NOTE: GDP = gross domestic product; KIS = knowledge-intensive services; RD = research and development; S&E = science and engineering; SS = social services; VA = global value added. SOURCE: Panel Analysis and S&E I 2012, see http://www.nsf.gov/statistics/seind12/tables.htm [November 2012]. Single Nation Analysis The clusters shown in Figures F-10 and F-11 are not surprising, as sector-specific expenditure variables are clustered together; i.e., business R&D expenditure figures are similar to each other irrespective of the data source. The same conclusion can be drawn for figures on expenditures on federal R&D, nonprofit R&D, and academic R&D. Eurostat, OECD, and UNESCO report numbers of FTE researchers for the United States, but it is not clear how that number is calculated. NCSES and NCES report head counts of S&E human resources. Therefore, a disparity is seen in the metric that is reported, as U.S. head counts represent the supply of human capital not necessarily involved in R&D. FTE researchers show the contribution of labor hours to the R&D process; hence it is important for a researcher or an S&T policy maker to understand that different data sources may appear to report the same thing, but this usually is not the case. Figures F-12 and F-13 show that variables representing head counts are not strongly correlated with FTE researchers. One advantage of having so many variables is that as a user, one can select among them depending on the question being addressed. PREPUBLICATION COPY: UNCORRECTED PROOFS

OCR for page 218
F-58 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION: IMPROVING INDICATORS TO INFORM POLICY Table F-5 shows that the whole set of variables produced by NCSES is an attempt at capturing different segments of the S&E population, which range from scientists to medical researchers. Figures F-12 and F-13 show that variables from NCSES and NCES are clustered together, with variables reporting the same indicator being more strongly correlated (see the cluster of doctorate recipients, graduate students [NCSES], and doctoral degrees [NCES]). This suggests the possibility that NCSES may be overproducing some of the S&E human capital variables. As previously mentioned, however, agencies produce variables to answer particular policy questions. The end result is a trade-off between efficiency and addressing user needs. It is commendable that NCSES has been able to satisfy academicians and policy analysts alike, but a more resourceful approach is required under current budgetary conditions. The panel would also like to highlight the efforts of NCSES to comply more closely with the recommendations of the Frascati Manual. The Survey of Industrial Research and Development (SIRD) (the old industrial survey) questionnaire contained items on FTE R&D scientists and engineers only. NCSES decided to resolve this data gap by including questions on researchers (FTE) and R&D personnel (head count) by gender; occupation (scientists and engineers, technicians, support staff); and location, including foreign locations. With the new data, it is possible to generate tabs, for example, on female technicians working in Belgium. The Survey of Research and Development Expenditures at Universities and Colleges (the old academic survey) contained a serious data gap in terms of information on R&D personnel in the academic sector. In 2010, NCSES began using the HERD survey to collect researcher and R&D personnel head counts. The HERD redesign investigation process indicated that collecting FTE data would be highly problematic, whereas collecting principal investigator data appeared to be rather reasonable. Therefore, obtaining information on FTE researchers or R&D personnel in the academic sector still is not possible, but one can obtain head counts of researchers, principal investigators, and R&D personnel. One point that came to the panel’s attention is that NCSES does not publish its main STI indicators on a single webpage. For national R&D expenditures, a user accesses National Patterns, while for human capital in S&E, one must generate tables from SESTAT. For further detail on academic R&D expenditures, WebCASPAR serves as a more useful tool. IRIS contains historical data tables on industrial R&D expenditures. SESTAT data feed into various NCSES publications, including (1) Characteristics of Scientists and Engineers in the U.S.; (2) Characteristics of Doctoral Scientists and Engineers in the U.S.; (3) Doctoral Scientists & Engineers Profile; (4) Characteristics of Recent College Grads; (5) Women, Minorities, and Persons with Disabilities in Science and Engineering; and (6) various InfoBrief’s. It is difficult to find summary tables that combine information across all five publications. WebCASPAR contains detail on SEH degrees that is not available in SESTAT. When staff of the Committee on National Statistics downloaded STI databases of other agencies/organizations, they had an easier task because all variables were available on a single webpage. PREPUBLICATION COPY: UNCORRECTED PROOFS