8

Feasibility and Implementation

Several prominent survey methodologists participated in a discussion on the feasibility and considerations for potentially implementing a national-level data collection on convergence. Graham Kalton (Westat) focused his comments on two areas of interest to the committee: (1) Relative to other concepts that we can measure in the context of existing science and technology indicators, and within current National Center for Science and Engineering Statistics (NCSES) surveys and reports, how valuable would it be to construct national-level measures of convergence or interdisciplinary research (IDR), and should these measures be sought in addition to existing measures or should they replace some existing measures? (2) What if any kinds of questions about the emergence of convergence/IDR are of greatest value to the nation and, in particular, which questions provide valid and actionable information?

Based on the workshop discussion to that point, Kalton observed that although the multidimensional aspects of convergence seem to be essential, they also create real challenges with respect to measurement in existing large-scale surveys. If one or two dimensions could be identified that could be used to provide imperfect indicators of convergence, though, it might be possible to collect data on a national scale that could be used to move the field forward. Because resources are limited, Kalton suggested that National Center for Education Statistics (NCES) might collaborate with other agencies such as NCES or the Bureau of Labor Statistics. He referenced the presentation by Anderson on the Business and Enterprise Research and Development (BERD) and Higher Education Research and Development (HERD) surveys. It also occurred to Kalton that another data collection vehicle that might be

amenable to some modification is the Survey of Doctorate Recipients (SDR), a panel survey that could be tapped to collect longitudinal information about activity in convergence research among doctorate recipients. Kalton acknowledged that the feasibility of producing one or two indicators must be addressed first. He wondered whether there was any evidence that another country has successfully applied this approach at a national level.

Kalton expressed concern about the quality of data extraction by nonspecialists to answer complex questions that incorporate cross-references to lengthy definitions of terms. He was skeptical that the data extractors would fully comprehend the definitions and then reliably search out the data, any failure to do so undermining the quality of the responses. Kalton thinks that the quality of the survey responses to complex questions on convergence needs to be carefully evaluated.

If one or two measures related to convergence were identified as useful indicators of the developments in this field, then this would raise questions about how they could be used to guide actions by the National Science Foundation (NSF) and others. Knowing which combinations of fields of expertise and in what settings convergence is more likely to occur would be valuable for targeting research programs. Kalton, however, remained pessimistic about the feasibility of identifying a small number of questions or concepts that could satisfy the needs of NCSES at this time. Although there is a high level of interest in convergent research, difficulty arises when applying the concept on a national scale.

Adopting an approach rooted in the traditions of the classical social sciences, Nora Cate Schaeffer (University of Wisconsin–Madison) began her remarks by asking what are the research questions to be answered, which is another way of asking how measures of convergence are to be used, and what is important for those measures to capture. For NCSES, it also means placing the definition and measurement of convergence within the context of the research enterprise. She followed with a number of other questions a social scientist routinely asks in the face of a measurement challenge: What is the underlying causal model of the construct? What units are involved in the causal process or causal processes? What characteristics of those units are important to measure, what units can be used for collecting data, and are the units from which data are collected and those for which measurements are needed different? Concerning the challenges presented in this workshop, Schaeffer asked, what unit of analysis does convergence belong to, what is its conceptual definition, and what is its operational definition?

To help explicate this approach more systematically, Schaeffer presented a tentative conceptual model of the social processes and actors that generate convergence (or not) (see Figure 8-1, showing units in yellow text, with some characteristics of those units noted in italics). This model depicts convergence as a property of research projects produced by researchers

within departments that are housed within institutions to which agencies make grants, although for some purposes, convergence might be conceptualized as a property of research teams or grants. The diagram is intended to illustrate some implications of clarifying how convergence is defined: To what unit should convergence be attached? Is convergence a property of a project? Is it a property of the goals of the project? Many different possibilities exist, and it is probably premature to settle on just one possibility. She added that the definition of convergence and the causal, social, and scientific processes underlying convergence may change as a field evolves, ages, and spreads around the world.

Schaeffer noted that funding agencies and their calls for applications could create incentives for institutions and researchers to pursue convergent research. Institutions are critical to this effort because they supply the infrastructure needed to conduct convergent research and the incentives to do so. The workshop participants heard absorbing discussions about the ways in which universities can foster interdisciplinary or convergent work, such as making bookkeeping and accounting systems communicate seamlessly across different units. These institutions are also the data collection units for the NCSES surveys that serve important roles in the measurement of

convergence. It therefore can be problematic if convergence is defined only at the level of projects, papers, discoveries, or patents but left undefined at the institutional or university level, especially when thinking about options available for implementing a definition or a measurement strategy. Schaeffer does not see a path toward a “unitary” definition of convergence that is relevant for all research goals or all units, but also does not consider a single definition to be necessary. Instead, she sees convergence as a concept with different meanings, different units, and activity at different points in time. If the measurement strategy remains flexible, then lack of a single definition for convergence is not necessarily a problem, as long as investigators specify which conceptual and operational definitions they are using.

Schaeffer mentioned three such concepts that are closely related to but distinct from the concept of convergence: interdisciplinarity, collaboration, and synthesis. If these are seen as dimensions of convergence, and each dimension is measured separately, the resulting data might be reliable and flexible, which could make it easier to monitor change over time. A single concept of convergence that is broad and vague is not necessarily plausible or necessary. If convergence is a property of projects, then measuring convergence at the level of the project may be reasonable. If, for example, researchers focus on interdisciplinarity, which seems to be one dimension of convergence, an operational definition that uses the standard departmental designations of the researchers on a project could be implemented at the level of projects. Information gathered for projects could be collected by institutions and aggregated to yield measures at a national level. Such measures could be more reliable than measures that ask investigators to assess the convergent qualities of their research projects.

She posed the possibility of measuring convergence at the level of the grant award. If funding agencies can agree on a definition of convergence and tag characteristics of a grant, then collecting and analyzing grant-level data should be feasible and potentially reliable. Another idea would be to continue the relationship with universities and support them in the development of infrastructure that allows for more straightforward implementation of measures of the critical concepts.

Andrew Zukerberg (National Center for Education Statistics) began his comments by focusing on the task of operationalizing convergence and its measurement. He highlighted the impact of convergence and the measurement of its impact on science and scientists. He asked whether investigations should have an indicator that serves as a baseline frequency that can be used as a benchmark for comparisons or a measure of the impact of the convergence. An indicator could focus on the frequency of convergent projects or the level of funding directed to convergent efforts. Zukerberg noted that impact was frequently mentioned and therefore should also be defined.

Zukerberg reviewed the definitions of convergence, beginning with the one provided by the National Academies in 2014.¹ He highlighted two primary characteristics within this definition: deep integration across disciplines and research that is driven by a specific and compelling problem. Adding a component that addresses improving quality of life would appropriately reflect the comments heard throughout the workshop. He agreed with Kalton and Schaeffer about the challenges of measuring an indicator that lacks a standardized definition. Still, he continued, unidimensional characteristics are easier to measure without a standardized definition in place. He acknowledged the issues inherent in providing a definition within a survey instrument. Even if an appropriate definition is included with the questionnaire, one cannot assume that the respondents will read the full definition and answer in the intended fashion.

Zukerberg offered a workaround to the definition issue using his research on bullying (which is based on Bureau of Justice Statistics data) as an example. When a concept such as bullying or stalking is presented, participants have immediate reactions. If instead individual components of a larger concept are presented, a scale can be built from the individual components in place of a standardized definition. For bullying, a standardized set of individual concepts might include answers to questions such as: Did someone exclude you intentionally? Did that person have more power over you? Did the event happen repeatedly? Along similar lines, if a standardized definition of convergence is not possible, measuring standardized concepts that make up aspects of convergence such as translational ability, diversity in disciplines, and a specific or compelling problem could help. Although it can be burdensome to divide many different concepts into parts, the individual components can be used to build a complete measure.

Zukerberg emphasized the need to consider the opportunity costs when adding a measure, which may require the removal of other questions from a study or replacing entire studies. One must consider whether the data provided by the new measures are valuable enough to justify the deletion of currently included scales and measures. Still, issues may arise in retrieving a complete dataset, particularly for an outcome aspect of research, because of the complexity of the underlying constructs. To understand questions regarding a larger-scale or long-term impact on researchers, NCSES could conduct a longitudinal or panel study (such as the Survey of Doctorate Recipients, or SDR) with measures based on its priorities.

___________________

¹ National Research Council. (2014). Convergence: Facilitating Transdisciplinary Integration of Life Sciences, Physical Sciences, Engineering, and Beyond. Washington, DC: The National Academies Press.

DISCUSSION

After the formal remarks, Jolene Smyth posed the question of whether attendees at the workshop think that it might be possible to identify one or two dimensions of convergence with a level of agreement that would warrant commencement. Jason Owen-Smith replied that efforts should be made to begin investigations, even if a perfect strategy has not been laid out. He believes that there should be a working definition paired with concrete examples and testing that could be revised in an iterative method, mainly because there are empirical components. Holbrook agreed with Owen-Smith and stated that an iterative experiment would be the best approach to take until consensus is reached on either a single definition or the most salient components. In his view, lacking a consensus should not inhibit efforts. Future progress can be expected to build on the initial efforts, react nimbly, and reverse course if necessary.

Schaeffer asked what unit would be the most useful for measuring convergence and whether the answer that researchers and NCSES give to this question differs. She elaborated that thinking about units would simplify thinking about the dimensions of a concept.

Hall raised the topic of measuring social impacts from the workshop’s first day. She added that considerations of social impacts should be carried out on a problem-specific level: identifying the degree of integration and variation of contributors (e.g., experts, patients, community stakeholders) represented in a publication or investigating preregulatory reports that influence policy regulation. An investigation of policy-impacting reports would signal the impact of research on the greater society.

Entwisle revisited Kalton’s initial question about why there is a desire to know more about convergence. Reflecting her inclination for a portfolio type of approach, Entwisle recognized that strategic investments in project areas are necessary and are an input to the research portfolio. Less clear are the outputs. In addition, as others have previously mentioned, there is interest to focus on impact as well as outputs. She added that there should be a clear distinction between the units of observation (e.g., funded projects, published articles, and investigators) and units of analysis (e.g., investigators, institutions, and fields) because this distinction will hold considerable implications for the data collection procedures. Smyth added that the determination of what will be the most critical issue to address is the core challenge.

Kalton shared concerns about the societal impact component of convergence ideals. For example, he mentioned the time delay between the conducting of research and the measuring of societal impact. Additionally, he had concerns about deep integration. To identify deep integration, one would need to obtain information from the project director or someone at

that level. In his own work as a survey researcher and as a biostatistician, he has often been involved in multidisciplinary research in which a team of researchers from different disciplines have collaborated to conduct a research project. He would not classify this team research as deeply integrated simply because it included researchers from multiple disciplines. He asked how one would measure deep integration and how to exclude the many cases in which the multidisciplinary efforts are standardized and not fully integrated.

Kalton asked if and how interdisciplinarity has been measured at a national level in other countries? Hall noted that in the coming weeks, she would be meeting with a group of scientists from the Swiss equivalent to the NSF at a workshop that would grapple with similar issues of defining and measuring convergent research. She is hopeful that an international effort might allow the countries involved to help one another and learn from each other.

Josh Schnell (Clarivate) had asked the group if determining what convergence is not might be easier than defining what convergence is. Smyth restated Michèle Lamont’s (Harvard University) research findings that convergence includes dimensions beyond a cognitive state such as emotional and interactive components. Elements of Schaeffer’s statements were also included, and Smyth reminded members of the difference between convergence and collaboration. Determination of the line where collaboration advances into convergence could become an indicator but the overlap between the two needs to be addressed in order to measure the advancement accurately.

Hall stated that there are different degrees of integration that would allow a distinction between terminologies to be made when measuring collaboration or team science. Her thoughts centered around NSF’s conceptualization of convergence and the levels of integration that might be expected or included in considerations of convergence. Smyth added that there might be unconventional ways to measure aspects that were previously unrealized as indicators of convergence and that inform a level of interactivity or collaboration occurring in research projects.

Schaeffer observed that interdisciplinarity may be less valid for some of the research purposes for which the concept of convergence is currently being used, but it might be more reliable because operational definitions could be based on an existing institutional infrastructure and a known collection of disciplines or departments. Though the infrastructure provided by a collection of disciplines is imperfect, it is reliable, flexible, and usable. She reminded attendees that emergent fields could become schools and departments within universities relatively quickly, as displayed by Dan McFarland’s (Stanford University) research. Using disciplines as a supplemental index could be useful to build a larger concept of convergence,

the full definition and explicit measurement of which currently eludes the research community.

Smyth shared comments made in the virtual chat where participants asked how many items could be included for measurement within a survey. Zukerberg stated that the number of inputs varies, particularly based on the context in which they are used. He assumed any measures of convergence would be a component of a larger survey, highlighting the need to balance the complexity of the survey. He added a question about the outcomes aspect and the size of the problem to be solved. Considerations of output will impact the unit of measure. He added that if a funding agency is providing resources, it implies that the problem has been deemed sufficiently necessary and other components should be weighted more heavily.

Erin Leahey (University of Arizona) mentioned Subject Categories in the Web of Science Core Collection—there are currently over 250 of them—and their potential use to advance convergence, particularly over time. According to Leahey, the list of subject categories changes quite regularly, even multiple times per year. There is no longer a single category for psychology, for example: It has been split into multiple types of psychology (developmental, clinical, and social). Changes in the Web of Science Subject Categories reflect areas that are merging, being excluded, or added over time, and this might be able to indicate the nature of convergent activity. Smyth added that the NCSES field-of-study categories for the dissertation question are another avenue for examining how fields have evolved over time.

Owen-Smith expressed surprise that the group’s discussion seemed to have quickly defaulted to measuring convergence through a sample survey using a multifactorial question structure. In his opinion, the group is not yet ready for that level of discussion. He asked the group if convergence could be identified on sight but not explicitly defined. Given the lack of a complete definition and in light of the nonsurvey data presented in the workshop, he believes it might be worthwhile to identify projects deemed to reflect successful convergence efforts. New research could follow successful approaches to work toward an empirically based set of metrics before developing a formal definition to be used in a national survey. Smyth commented that perhaps we need a convergent approach to studying convergence.

Hall responded to Owen-Smith’s comments and reiterated that the distance between disciplines and the numbers of disciplines involved are critically important. There is, for example, a large difference between a research project that integrates developmental and social psychology with health psychology versus a project involving a social psychologist working with a physicist. The differences in perspective on what constitutes interdisciplinarity will depend on the individual researchers’ orientations and the intellectual perspectives they are accustomed to within their own disciplines. Experts working in disciplines and organizational structures

that do not tend to collaborate broadly (e.g., due lack of rewards, recognition for cross-disciplinary collaboration via tenure and promotion reviews) are likely to face more barriers and have less experience considering how they might apply their knowledge to a broader problem or communicating how collaborators outside their discipline may integrate their specialized knowledge when addressing common problems.

Smyth built on these points to remark that if a measure is to be included in a survey for humans to answer, it is essential to remember that the respondent’s understanding of the topic will inevitably influence how they answer the questions presented. She encouraged the group to consider how people may have different understandings of interdisciplinary research. Smyth noted that it may be premature to consider adding measures to a sample survey because it could have unintended consequences.

This page intentionally left blank.