Enhancing the Effectiveness of Team Science (2015)

10

Advancing Research on the Effectiveness of Team Science

The committee’s review of the research related to the study charge yielded many new insights into approaches to enhance the effectiveness of team science. However, it also identified gaps in the evidence base where further research is needed. Here, we discuss some of the research needs in greater detail and the promise of new methods for use in addressing them.

TEAM PROCESSES AND EFFECTIVENESS

As discussed in Chapters 1 and 2, science teams and larger groups share many challenges with teams in other contexts, and, thus, the research on teams in other contexts is relevant for team science. In particular, team process factors, such as the development of shared understanding of team goals and roles, have been shown to influence the ability of teams to achieve their goals, both in science and in other contexts. Drawing on this research, previous chapters recommended actions and interventions in three aspects of team science—composition, professional development, and leadership. At the same time, however, we have noted the need for further “basic” research on team processes within science teams and larger groups and how these processes are related to scientific discovery and translation.

Improving an understanding of the processes of team science will require interdisciplinary collaboration involving experts in the various disciplines that study teams and organizations (i.e., psychology, organizational behavior, communications) and in the science of team science and related fields (such as economics, science policy, philosophy of science and sys-

tems science), along with team science practitioners. Investigators working together could develop a comprehensive, multi-method measurement approach to investigating the dynamics and outcomes of science teams and larger groups. Such an approach includes, but is not limited to, bibliometric indices, co-authorship network analyses, experts’ subjective appraisals of team science processes and products, and surveys and interviews of team science participants. In particular, valid and reliable metrics are needed to more clearly understand the process of deep interdisciplinary knowledge integration and how it varies in unidisciplinary, multidisciplinary, interdisciplinary, and transdisciplinary science teams and groups (Wagner et al., 2011). Along with advances to metrics, investigators working together can apply rigorous experimental methodology (e.g., manipulations, control conditions, before–after data) to science teams and groups to develop a deeper understanding of causal mechanisms underlying effective team science.

Future efforts to understand team science processes can be aided by new approaches, such as the complex adaptive system approach discussed in Chapter 2. In addition, new data collection methods are becoming available, such as the use of wearable electronic badges that unobtrusively trace scientists’ interactions as they work (see further discussion below). This research should use methods sophisticated enough to address longitudinal changes across levels of analysis (e.g., individual, team, organizational) and the resulting mediators and moderators of the hypothesized effects; such methods are described in the final section of this chapter.

Specific research gaps associated with science team composition, professional development, and leadership are highlighted in the following three subsections.

Team Composition and Assembly

In Chapter 4, we concluded that methods and tools that allow practitioners to consider team composition systematically appear promising and recommended that those involved in assembling science teams and larger groups apply these methods and tools. As team science leaders begin to apply task analytic methods to compose science teams and larger groups (implementing Recommendation #1 in the Summary), evaluation studies are needed to guide refinements and improvements to these applications. An ongoing cycle of implementation, evaluation, and revision would further strengthen the ability of team and group leaders to identify the task-relevant diversity needed to achieve the scientific or translational goals of the project. Chapter 4 also discussed recent research on the team assembly process. Further research on the assembly process in science teams, including comparative studies of the processes and outcomes of self-assembled versus assigned teams, would provide valuable information to the scientific

community, funding agencies, and university administrators. Studies on the implementation and impacts of the new research networking tools that are being adopted by many research institutions would also be valuable.

At the same time, Chapter 4 highlighted the disagreements and uncertainties in the research to date about how various individual characteristics may affect team outcomes. In light of these uncertainties, there is a clear need for further and more sophisticated research on how the multiple individual characteristics of the team or group members combine within science teams and groups, and how these interactions and processes are related to effectiveness. This research would address such questions as:

• What is the role of individual characteristics (including dispositional qualities such as social intelligence) in team processeses and effectiveness?
• How do the interactions among subgroups (whose members may share multiple similar characteristics) affect team processes and effectiveness?
• How does team composition interact with team processes to influence team effectiveness?
• How do changes in science team or group membership impact team processes and outcomes?
• How may the various roles team or group members play (e.g., connectors/brokers, leaders, scientists with particular expertise, community stakeholders) be characterized? What are the interrelationships between these roles, and how do they affect team processes and effectiveness?

Professional Development and Education for Team Science

In Chapter 5, we concluded that several types of professional development show promise to improve the processes and outcomes of science teams. As universities, researchers, and practitioners begin to create professional development opportunities for science teams (implementing Recommendation #2 in the Summary), ongoing evaluation of these opportunities would provide valuable information for continuous improvement of them. In addition, more basic research on how science teams and groups learn and develop would enhance future professional development.

We also concluded that colleges and universities are increasingly developing cross-disciplinary programs designed to prepare students for team science, but that little is known about the effectiveness of these programs. In particular, we noted that some of these programs do not clearly articulate the competencies they are intended to develop and they target a variety of competencies. The literature has produced a plethora of competencies that

overlap to some degree and also have differences. And, little empirical research is available on the effectiveness of such programs in developing the various competencies that they target. Methods used to date to evaluate these programs rely heavily on case studies and expert reviews.

Addressing these gaps in the research evidence will require collaboration between the multiple communities engaged in interdisciplinary education and the team-training research community. Through such collaboration, researchers could create methods for assessing both collaborative and intellectual outcomes to identify core competencies that could then be systematically integrated into graduate and undergraduate programs to prepare students and team members for team science. More generally, collaboration among these communities would make it possible to conduct more robust prospective studies that compare and explicitly evaluate the relative effectiveness of various educational programs designed to prepare students for team science. In particular, it will be important to address the following unanswered questions:

• How is variation in the competencies developed through education and/or professional development related to team science processes and outcomes? For example, under what conditions do teamwork training (focused on team-related knowledge and skills) and task work training (focused on the scientific knowledge and skills) enhance scientific productivity?
• What educational or professional development approaches are most effective in developing the targeted competencies at different educational and career levels (e.g., doctoral education vs. senior investigator)?

Team Science Leadership

In Chapter 6, we concluded that 50 years of research on team and organizational leadership in contexts other than science provides a robust foundation of evidence to guide creation of leadership development programs for team science leaders. As researchers and team science practitioners begin to develop such programs (implementing Recommendation #3 in the Summary), ongoing evaluation is needed to inform continued revisions and improvements. An ongoing cycle of continuous improvement, based on testing and evaluating the new courses, would enhance the quality of future leadership development programs for team science. Such efforts would enhance participants’ capacity to lead in ways that facilitate positive team processes and enhance scientific and translational effectiveness. At the same time, more basic research could guide these efforts by, for example, investigating the applicability of promising recent leadership approaches to

science teams and larger groups, including contextual leadership, emergent leadership, team leadership, and shared leadership.

SUPPORT FOR VIRTUAL COLLABORATION

In Chapter 7, we concluded that when scientific colleagues are geographically remote from one another, issues such as lack of shared vocabularies and experiences and role confusion may be exacerbated relative to face-to-face teams or groups. Although the research supports our recommendation that team leaders take several steps to address these issues, it would be valuable to conduct further research on the extent to which the research on teams and groups and principles for effectiveness identified in Chapters 3 through 6 are applicable to virtual science teams and larger groups.

We also concluded that technology for virtual collaboration often is designed without a true understanding of users’ needs and limitations and may thus impede such collaboration. Hence, further research is needed to evaluate how tools and practices for virtual collaboration affect team processes and outcomes. This requires that researchers, technology developers, and technology users work together to conduct research on user-centered design and human-systems integration so that the various tools for collaboration are interoperable and are aligned with users’ activities and capabilities.

INSTITUTIONAL AND ORGANIZATIONAL SUPPORT FOR TEAM SCIENCE

In Chapter 8, we observed that many universities are launching efforts to promote and support interdisciplinary team science, but research is sorely needed to guide these efforts, so that they succeed in fostering team science and advancing scientific discovery and translation. To date, the impact of these efforts on the amount and quality of team science research remains to be systematically evaluated. In particular, we noted that university-industry research collaborations have grown faster than the knowledge of how to manage them effectively. Limited systematic, rigorous research is available on such partnerships, and there is a dearth of research on failed collaborations. In addition, we noted that research on the relationship between design of the built environment and scientific collaboration remains theoretically debated and empirically mixed. Some studies have found a positive relationship between spatial proximity and scientific collaboration, but additional research is needed to improve understanding of the relationship between the design of the built environment and team science effectiveness. A broader focus for this research would examine cultural

and social factors intertwined with the spatial environment that may jointly affect collaborative processes and outcomes.

A few studies are beginning to examine some specific university strategies to promote interdisciplinary team science. For example, one recent study examined how Harvard Medical School’s “open” call for research ideas aided development of research topics on Type I diabetes (Guinan, Boudreau, and Lakhani, 2013). The committee encourages more agencies and universities to study and learn from existing and emerging strategies to enhance the way science is supported and conducted.

A follow-on study to the 2005 National Academies survey of institutions and individuals conducting interdisciplinary research (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 2005) might be a helpful step in guiding university efforts.¹ The findings would illuminate what progress has been made in the past decade, what obstacles still remain, and what research-based promising practices can be identified. In turn, the results of this new, follow-up study could be used in formulating more specific research studies to increase understanding of how alternative types of organizational and institutional policies and practices affect team science.

More generally, research on university efforts would provide greater clarity if it included more field and quasi-experimental study designs with longitudinal and panel components to examine the outcomes of university efforts over time. Studies of university-industry partnerships and other multi-stakeholder team science projects are needed to examine choices of institutional partners, factors related to both success and failure of these projects, formal and informal management practices, and the nature of their institutional impacts. Such studies would benefit from the development of data collection strategies and a performance data system that is transparent, meaningful, and accessible to researchers.

In Chapter 8, we also noted a few, isolated examples of university efforts to change policies and practices related to awarding credit for team science in the promotion and tenure process. Despite such exceptions, university policies for promotion and tenure review typically do not provide comprehensive, clearly articulated criteria for evaluating individual contributions to team-based research. Recognizing that disciplines, departments, and universities will continue to establish and apply their own criteria for evaluating research contributions, we recommended that universities and disciplinary societies proactively develop broad principles for assigning individual credit for team-based work. Targeted research is needed to inform these efforts, along with research on the feasibility and effectiveness of providing team rewards (e.g., bonuses, public recognition) for team-based work.

______________

¹Such a study need not be conducted by the National Academies.

More generally, research is needed to increase understanding of the promotion and tenure process as it relates to team science. A valuable first step would be a systematic survey of U.S. universities’ promotion and tenure policies related to evaluating individual contributions to team-based research. The limited information currently available suggests that such policies include a relatively narrow range of criteria relative to the broad range of potential meaningful contributions an individual can make to a science team. Further research is needed to develop evidence-based principles for evaluating contributions such as being a “broker” who brings individuals and/or organizations together (a role that has been shown to facilitate innovation as discussed in Chapter 4).

In addition, research is needed to understand how such new principles and criteria could best be implemented, addressing such questions as:

• To what extent are university-wide policies implemented and adhered to?
• What factors, such as university, school, or departmental leadership and culture, influence the uptake of new policies?
• How long does it typically require before policy changes affect practice within promotion and tenure committees?

Research is also needed to explore team rewards for team science. Although many members of science teams and groups work at universities, others are found in industrial research and development laboratories, freestanding science facilities (e.g., particle accelerators or large observatories), federal laboratories, and public and private research centers and institutes. Regardless of where they are employed, scientists and other stakeholders engaged in collaborative research may respond to incentives and rewards provided by their employers. To date, despite the rapid growth of teams in science and other sectors of the economy, organizational incentive systems have focused primarily on rewarding individual achievements. Further research is needed to develop and test team-based rewards for team-based accomplishments. Such research would benefit from a collaborative approach including organizational scientists who have begun to examine team rewards in other contexts as discussed in Chapter 8 (e.g., Chen, Williamson, and Zhou, 2012) and experts in the science of team science.

Finally, we noted in Chapter 8 that there is a general lack of research on team science from an organizational perspective. Further research from this perspective would be valuable to inform research and practice. For example, the emergence of such new organizational forms as multi-team systems, cross-network scientific collaborations, and large, geographically dispersed research centers may require new approaches to team or group composition, professional development, and leadership. However, we noted

in Chapter 6 that there has been little research to date on leadership in multi-team systems; only a few studies have begun to explore how system and team leaders can best foster coordination within and among the component science teams. Similarly, new organizational forms of team science are likely to present new challenges for composing and assembling the team or group, and for providing professional development.

FUNDING AND EVALUATION OF TEAM SCIENCE

We have noted that evaluating the processes and outcomes of team science is challenging, in part because science teams and larger groups may have multiple goals. Research is needed to develop new evaluative criteria that are appropriately matched to the respective goals and concerns of the teams, groups, organizations, institutions, funders, and community groups that have a stake in the foci, processes, and outcomes of the projects. In Chapter 9, we noted that federal scientific agencies are increasingly interested in examining their own processes, so that they can improve their practices and better address important social, technical, and scientific challenges. To date, however, very little empirical evidence is available from such efforts. Research is needed to help both public agencies and private foundations best deploy their resources to foster effective team science and find the optimal balance between team and non-team approaches. This research would provide answers to questions such as:

• How can funders and scientific review panels better identify team proposals that are likely to succeed or fail?
• What happens when the funding for a science team or group is withdrawn? Does the lack of long-term funding commitment lead researchers to revert to more traditional small, incremental scientific development processes? What is the relationship between the sustainability of funding and a supportive institutional context in terms of the likelihood of long-term success?
• What types of management, beyond the traditional funder roles of evaluating research proposals and requiring written reports, might facilitate science team effectiveness?
• Would team effectiveness be enhanced if funders provided ongoing technical assistance and emergency assistance to address collaboration challenges as they arise?

More specifically, research is needed to understand how alternative funding strategies may affect science team effectiveness. In Chapter 9, we recommended that funders require collaboration plans. Studies comparing the effectiveness of teams and groups that did and did not include a collabo-

ration plan in their proposals would enable a learning and improvement-oriented approach to the management of team science.

Because peer review panels function as teams in and of themselves, research to better understand how their structure and dynamics influence reviews of team science proposals would provide useful information to funders. It would also be valuable to study how new approaches in which reviewers assemble “dream teams” with the goal of rapidly advancing science and translating discoveries in targeted areas affect the processes and outcomes of these teams.

SUMMARY, CONCLUSION, AND RECOMMENDATION

In closing, we note the promise of new research methods and approaches for advancing the research on team science effectiveness. In Chapter 2, we discussed the unique concerns of the science of team science, including its focus on highly diverse units of analysis, ranging from the individual to the team, the organization, and society as a whole and the need for developing valid, reliable metrics and criteria to understand and evaluate team processes and their relationships to scientific and translational outcomes. We noted that new research approaches and methods could help the field with these various concerns. For example, complexity theory offers a promising route to understand how behaviors, actions, and reactions at each level of a system affect actions at the other levels and the emergent behavior of the system as a whole. Researchers have begun to investigate team science using a complex adaptive system approach.

New methods are also available for studying team dynamics. For example, team or group members can be equipped with small electronic sensor badges (about the size of a smartphone) to record data on their interactions,

including whether they are face-to-face, how close they are to one another, and the intensity of their conversation. Similarly, electronic communication data, such as emails and texts, can be recorded and analyzed. Data illuminating team or group dynamics—whether captured by unobtrusive sensors, through records of electronic communications, or through more traditional surveys—can be creatively combined with bibliometric data to examine the relationship between team processes and outcomes (in the form of scientific publications). Because team or group dynamics, goals, and outcomes change over time as science teams move through different phases in their work, longitudinal research designs coupled with analysis of temporally tagged data can provide greater insight than cross-sectional, one-time approaches.

Empirical research on science teams and groups can also benefit from simulation and modeling methods. Simulation allows technological tasks conducted by science teams and group in the real world (e.g., joint use of scientific equipment or virtual meeting technologies) to be studied under controlled laboratory conditions. In this way, technologies can be evaluated on the basis of their ability to improve science team effectiveness. Also, computation models (e.g., agent-based models, dynamical systems models, social network models) of findings regarding team member interactions under varying conditions in the literature on teams (including science teams) can help to extend empirical results from small science teams to larger groups and organizations.