National Academies Press: OpenBook

Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop (2018)

Chapter: 5 How Can Analytics Be Used to Make Decisions About Adaptability?

« Previous: 4 What Does Adaptability Look Like?
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

5

How Can Analytics Be Used to Make Decisions About Adaptability?

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

In the workshop’s third panel, moderated by Annette Parker, president of South Central College, presenters described methods to analyze and shape adaptability and programs designed to foster adaptability. For example, labor market data can help organizations and individuals understand their environments to better facilitate adaptation, and measures of individual and organizational adaptability can guide decisions and future actions.

A DATA AND INNOVATION HUB

The object of RealTime Talent, explained Executive Director Sandee Joppa, is to enable more informed, market-oriented decisions throughout the Minnesota workforce and education ecosystem by engaging a broad group of stakeholders. The organization works with employers and employer associations in key industries in the state, large higher education institutions—including Minnesota State University and the University of Minnesota—and the state government. It “brings people to the table to [explore] what are we doing about supply and demand, what are we doing about the workforce shortage, how might we make better decisions, and [how can we] take better actions if individual constituencies are informed.”

Minnesota has strengths and weaknesses in developing, recruiting, and retaining talent, she said. Employers represent diverse and vibrant industries that provide historically well-paying jobs—Minnesota ranks 13th in the nation in income per capita and 2nd in the Midwest. The overall workforce is well educated and trained, with high participation rates compared with national averages. In the near and long term, demand for workers across skill sets is expected to continue to grow, at 1.5 percent annually across all sectors. However, workforce growth will slow in the near term as current workers age and the number of new workers declines. Net out-migration will prevent the state from augmenting its slow workforce growth—about 6,000 more workers leave the state than migrate in each year. And the impact of employment disparities across racial and ethnic groups will grow as the workforce becomes more diverse, with the population of people of color expected to grow by 50 percent over the next 20 years. Given these trends, the state will have a projected worker gap of 287,000 people by 2022, with about half of that shortage in the Minneapolis–St. Paul area.

To illustrate the innovative approaches that RealTime Talent is bringing to Minnesota’s labor market, Joppa reported that the organiza-

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

tion has been working with the state legislature to bring a new job posting system to the state. After evaluating 25 online job posting systems, it selected one that algorithmically matches a person’s interest with what a job requires. The matching algorithms are blinded to remove bias that might be introduced, for example, by where job seekers went to college, where they worked last, and what their last names are. The state is also seeking to eliminate disparities in employment by raising the labor force participation and employment rates of all racial and ethnic groups to match or exceed those of native-born whites, increase domestic migration to a net positive of 5,000 people per year, and maintain international immigration rather than letting it slow. If these steps are successful, they could reduce the state’s worker gap by almost a third, to about 200,000, Joppa said.

RealTime Talent has divided the state into regions, each with its own characteristics. For example, the economy of the northeastern part of the state is based on mining, logging, and paper and apparel manufacturing; average income is low and unemployment is high. The southeastern part of the state is focused on textile and food manufacturing and has low unemployment and a low average income. These regional differences are a key factor in the state’s educational system, particularly the Minnesota State system, which offers both two- and four-year degrees. “We are getting employers, educators, and other people together at the table to help solve problems in and create opportunity in those geographies,” Joppa reported.

A popular product of RealTime Talent’s work, according to Joppa, is its one-page summaries of job data in a region. In the Twin Cities area, for example, the jobs most in demand are registered nurse, customer service representative, administrative assistant, project manager, and business analyst.

This analysis also yields information on the top ten foundational “agility skills” that are in demand (figure 5-1). Skills shown in a lighter color in the right-hand column are specific to engineering/technical occupations compared to all occupations.

These kinds of data can help people take advantage of workplace opportunities, said Joppa. “We are always on the lookout for a new data source, a new perspective, or a new technology that we can bring to the state.”

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Image
FIGURE 5-1 Lists of skills in highest demand by Minnesota employers. Source: RealTime Talent, based on aggregate job posting data from TalentNeuron Recruit (www.wantedanalytics.com), accessed 10/18/2017.

ASSESSING ADAPTIVE PERFORMANCE IN THE WORKPLACE

The changing work environment is increasingly demanding an adaptive workforce, observed Tracy Kantrowitz, director of talent solutions at PDRI. She reported the following indications that change is frequent in organizations and increasingly requires employees to adapt to new situations:

  • Employees have greater interdependence and now work with an average of ten people to get a job done.
  • Organizations change frequently, with the average employee experiencing some form of organizational change—such as a change in leadership, a merger or acquisition, or a restructuring—every seven months.
  • Increasing numbers of jobs are knowledge based, with 82 percent of employees doing work that requires analysis and judgment.
  • Companies are often geographically dispersed—the amount of work done with coworkers in another geographic location has increased 57 percent in just the past three years.
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
  • The demographic profile of employees is changing; as baby boomers retire and a new generation enters the workforce, employee work preferences are changing.

Adaptability is a multidimensional concept that varies from one job to another. An air traffic controller may have to deal with crisis or emergency situations on the basis of real-time information. An executive assistant may need to handle unpredictable or uncertain circumstances. Engineers need to keep abreast of new tasks and technologies and procedures for working and solving problems creatively. “These all call for different kinds of adaptability,” said Kantrowitz.

In addition, many employees are encountering new and intensified attributes of the modern work environment. They need to handle work stress as companies strive to do more with less. “Do they remain composed or do they let that derail their performance?” asked Kantrowitz. “This is a type of adaptability.”

Many jobs require creative problem solving, working across teams, and a level of comfort in working with multidisciplinary teams to arrive at breakthrough innovations. Employees also need to learn new technologies and procedures. “How do people stay abreast of new knowledge and methods once they complete their formal education? How do they adapt to new information? How do they continue to grow and develop in their careers?”

Interpersonal adaptability in how people communicate, approach others, and tailor their messaging to different stakeholders is a critical feature of many jobs. Cultural adaptability ensures that people are able to work as a team with people who are from different backgrounds and have different values. Some employees need physical adaptability to work in certain jobs.

Given the many forms of adaptability, there is no single way to measure this attribute, said Kantrowitz. Instead, it needs to be measured across the employee lifecycle, from the determination of adaptive performance requirements to selection of more adaptable employees to management of adaptive performance.

Kantrowitz described the Job Adaptability Inventory (JAI) as a method to determine the adaptive performance requirements of a job (Pulakos et al. 2000) and identify which dimensions of adaptability are most relevant. The results are used to determine the selection of an appropriate individual assessment to identify which people may be more predisposed or more likely to perform well in situations that require

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

adaptability. The result is a more holistic picture of adaptability rather than a single measure.

To illustrate application of the Job Adaptability Inventory, Kantrowitz picked four jobs and analyzed how the adaptability requirements differ among them (figure 5-2). Assessments of cognitive aptitude are based on abilities such as solving problems creatively, learning new tasks and procedures, and coping with uncertain and unpredictable work situations. Assessment of noncognitive traits is useful in identifying individuals who are more likely to perform well in jobs that require interpersonal adaptability, cultural adaptability, and handling work stress. And assessment of physical adaptability might look at a person’s capacity to handle emergencies and physical tasks.

Other measures of adaptability include past experience with adapting, interest in adaptive situations, and self-efficacy to adapt (Pulakos et al. 2002). These measures result in significant prediction beyond cogni-

Image
FIGURE 5-2 Sample assessment using the Job Adaptability Inventory, showing that adaptability requirements vary by job. Source: Pulakos et al. (2000).
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

tive ability and personality, and past experience is the best predictor, said Kantrowitz.

She also pointed out that companies are often interested in whether teams can be adaptive, in which case they may use individual adaptability measures and team variables to forecast how well the teams are likely to perform adaptively (Pulakos et al. 2015).

Finally, Kantrowitz noted the importance of training adaptable leaders (Mueller-Hanson et al. 2005). Training should incorporate many opportunities for emerging leaders to be exposed to situations requiring adaptability so they have a catalogue of experiences on which to draw. An iterative process of practice, feedback, and practice is necessary.

MEASURING ADAPTABILITY

Susan Straus, senior behavioral scientist at RAND, described a RAND evaluation of a course that the US Army developed to train adaptive leaders (Straus et al. 2014). The ten-day US Army Asymmetric Warfare Adaptive Leader Program (AWALP) was based both on the Individual Adaptability Theory (I-ADAPT; developed by Pulakos et al. 2000) and on outcomes-based training and evaluation. It was for noncommissioned officers and junior-level commissioned officers, with low student-instructor ratios thanks to small groups of students and larger groups of instructors (called guides). Unlike most army training, which is classroom based, this course had a small amount of classroom time and a number of practical experiences, such as problem solving and teamwork.

As mentioned earlier by Kantrowitz, the I-ADAPT approach has eight dimensions of adaptability, characterized as core, supporting, or enabling:

  • handling crisis and emergency situations (core)
  • handling stress (core)
  • thinking creatively (core)
  • dealing with changing or ambiguous situations (core)
  • interpersonal adaptability (supporting)
  • cultural adaptability (supporting)
  • physical adaptability (enabling)
  • learning tasks, technologies, and procedures (enabling).

Most of these dimensions are intangible, Straus pointed out, which means it can be difficult to assess them. Self-report and observational

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

measures can be time consuming and are subject to various biases and threats to validity. “Many leaders already think they are adaptable,” she explained, “so if you ask them how adaptable they are, they will say they are very adaptable.” Thus they have little room to improve on self-reports, and courses tend to show little change.

The evaluation used multiple measures and methods (described in Alvarez et al. 2004). For example, most course evaluations are done with an end-of-course survey asking participants what they got out of the course, but “what people think of the course and whether they learn are two different things,” Straus clarified. The course evaluation measured changes in knowledge, attitudes, and behavior and whether learning was transferred to performance that made a difference on the job.

An objective measure of transfer is very difficult, according to Straus; the measure used was one of perceived rather than objective transfer. Surveys measuring AWALP students’ reactions to the course used both closed and open-ended questions. Examples of the former included the following statements:

  • AWALP guides were knowledgeable about the subject matter.
  • AWALP guides effectively facilitated after-action reviews and group discussions.
  • The feedback I received from AWALP guides enhanced my learning.
  • Course materials supported the learning objectives.

Examples of open-ended questions were:

  • What did you like best about AWALP?
  • What aspects of AWALP should be changed? How would you change them?
  • Will AWALP change the way you lead others? If so, how?

Changes in the learners’ declarative knowledge were measured using multiple choice pre- and posttraining tests. For example, one of the 30 questions was:

The best definition of adaptability is:

  1. Having the capability to complete something in a different way than you have in the past
  2. An effective change in behavior in response to an altered situation
  3. Constantly changing to keep the enemy off balance
  4. Being able to effectively respond to crisis or emergency situations
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

Changes in the learners’ attitudes about adaptability were measured using a pre- and posttraining survey and a multidimensional approach, with items assessing students’ experience and need to be adaptable on the job, interest in engaging in adaptive performance behaviors on the job, and self-efficacy for the behaviors. The survey focused on six of the eight dimensions of adaptability emphasized in the course: thinking creatively; interpersonal adaptability; cultural adaptability; learning tasks, technologies, and procedures; handling ambiguity; and decision making under stress. Questions were based, in part, on Pulakos et al. (2000, 2002) and Ployhart and Bliese (2006). Figure 5-3 shows a question about dealing with ambiguous situations.

The survey also included questions about putting adaptive performance behaviors into practice in terms of managing others (based on White et al. 2005). Behavioral learning was measured with student and guide ratings of team performance in several practical exercises. Raters evaluated the degree to which the exercises required different dimensions of adaptive performance and how effectively the teams performed the behaviors. Finally, the evaluation collected data on individual differences that are likely to predict adaptability, such as openness to experience, learning goal orientations, and motivation for training.

The perceived transfer of training was assessed by talking to the graduates three and six months after they completed the course. Evaluators also sought to talk with the students’ supervisors, which was challenging because many supervisors did not respond to requests for a discussion or the students wanted to protect the supervisor’s time and did not respond to requests for their supervisor’s contact information.

Image
FIGURE 5-3 Sample survey question probing respondents’ experiences in dealing with ambiguous situations. Source: Susan G. Straus, “Measuring Adaptability,” RAND, presentation at the National Academy of Engineering workshop, November 2017.
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

“That may be more specific to the army than to other organizations, but it was a definite challenge,” said Straus. Examples of the questions were:

Have AWALP graduates changed professionally as a result of training? Do they do any of the following more than they did before the training?

  • Mentor and train subordinates
  • Delegate tasks
  • Seek input
  • Conduct after-action reviews
  • Brief commander or senior leader

For graduates, have your attitudes about AWALP changed 3 months and 6 months postgraduation?

  • Recommend to others
  • Recommended course changes
  • Challenges in applying concepts

The evaluation showed a convergence of results providing strong evidence for the success of the training, said Straus. The students had very favorable attitudes toward AWALP, which were sustained over time. They showed large increases in declarative knowledge about adaptability, much higher self-efficacy at the end of the course than at the beginning, greater interest in being adaptable, improvements in judging the need for adaptability, and, according to both supervisors’ and self-reports, greater awareness of the applicability of the course’s principles on the job. Straus noted that the question of how to measure the transfer of training to performance presented the greatest challenge to the study.

Straus concluded by identifying some ways to improve the assessment of such training. Online data collection—for example, using mobile devices—could facilitate analysis and feedback. Better training of instructors in rating team performance could improve the reliability of those ratings. The students’ work supervisors could be better held accountable to provide feedback on posttraining behavioral change. Longer-term impacts could be assessed through measures of the retention of course knowledge and attitudes, 360-degree feedback, measures of the performance of graduates’ teams, and—what Straus called her “holy grail”—randomized controlled trials.

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×

AVAILABLE MEASUREMENT TOOLS

During the discussion period, Kantrowitz and Straus elaborated on their use of the Job Adaptability Inventory and other measures of adaptability. The JAI is a proprietary tool, Kantrowitz pointed out, but other measures are available from public sources and through psychometric testing providers. And because assessment science has progressed in recent years, better tools are available, such as an exercise that puts people in simulated situations to evaluate the extent to which they are able to adapt in desirable ways. Such tools, which are available online, can measure both whether people are predisposed to react in certain ways and whether they have the knowledge and skills to do jobs that require adaptability. Straus added that many of these tools are essentially generic, so they can apply to any educational context, not just a particular course for army officers.

But, Straus cautioned, this work is in its early stages and best practices or best instruments still do not exist. Kantrowitz concurred, noting that good work is being done “in pockets” but that interest in adaptability has not caught on in many companies and industries. “It is not deeply entrenched in a lot of companies’ talent management programs, which is too bad, because it is clearly [a] pressing” need. Nick Donofrio observed that many companies have management programs, but they are mostly for specific purposes and have not been publicized.

Straus noted that much of this work on evaluating adaptability has been done in assessment centers, which offer a strong approach that has been used in many domains. “But,” she added, “it is also a costly approach, [which] is one of the reasons it is not more prevalent.” Perhaps the advent of computerized tools will make the assessment of adaptability less costly and more prevalent, she said. Joppa observed that much of this assessment work has been based on trial and error, since not enough is yet known about the role of adaptability in moving an employee from one job to another within a company.

Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 34
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 35
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 36
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 37
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 38
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 39
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 40
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 41
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 42
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 43
Suggested Citation:"5 How Can Analytics Be Used to Make Decisions About Adaptability?." National Academies of Sciences, Engineering, and Medicine. 2018. Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25016.
×
Page 44
Next: 6 Training and Organizational Change »
Adaptability of the US Engineering and Technical Workforce: Proceedings of a Workshop Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Late last year, the National Academy of Engineering (NAE) convened a workshop on Preparing the Engineering and Technical Workforce for Adaptability and Resilience to Change. The workshop springs from the earlier NAE report Making Value for America which described the ongoing transformation in the way in which products and services are conceived, designed, made, and distributed. The workshop focused on the challenges facing the workforce in light of these dramatic changes in the production process, especially the need to constantly renew and learn new skills.

The workshop served to increase stakeholders' understanding of both the importance of workforce adaptability and the definition and characteristics of adaptability. It also provided an opportunity to share known best practices for fostering adaptability, including identification of barriers and multiple pathways for overcoming those barriers. As important, it helped to identify needs for future study and development. This publication summarizes the presentations and discussions from the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!