National Academies Press: OpenBook

Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report (2016)

Chapter: 4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions

« Previous: 3 Organizational Strategies and Processes for Enhancing U.S. Air Force Communication Effectiveness
Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×

4

Measuring Communication Effectiveness Across Diverse Backgrounds and Missions

As with any program, measuring the effectiveness of communication is vital to understanding its value and learning how to shape it for greater effectiveness.

Brig. Gen. Kathleen Cook expressed to the committee a desire for definable, measurable, results-based outcomes, using the approach that “we are trying to change the knowledge, attitudes, or beliefs of a specific audience to achieve a specific action.” This clarity, she said, would not only inform the message, but also provide a construct under which to align the U.S. Air Force’s (USAF’s) weight of effort and inform what they should be measuring themselves against.

DEFINING “EFFECTIVE COMMUNICATION” IN THE AIR FORCE CONTEXT

What is “effective” communication? Akin to weapons deployment, it can be described as communication that reaches its target, impacts its audience, and achieves the intended objective for that particular audience. Several participants stated that whether in the context of an individual message or an Air Force-wide communication strategy, this requires setting objectives, planning, and acting in a deliberate and organized fashion.

The discussion among several of the workshop participants highlighted that in the process of planning the elements of a communication strategy, leadership should first define what they hope to achieve. These participants acknowledged that communication requires measurement, and the selected measures must be specific, measurable, attainable, relevant, and timely. To that end, they noted, leadership needs a clear and consistent vision of what success looks like; this vision of success should be tied to goals.

Ms. Katie Paine, chief executive officer of Paine Publishing, established the context for discussing this topic at the workshop by summarizing the Barcelona Principles, a set of industry standards for measuring communications effectiveness and outcomes, developed by an assembly of communication-measurement professionals in Spain.1 A few workshop participants discussed that an Air Force communication “baseline” could be defined as instituting within the communication culture a sense that the Air Force must measure its communications in order to understand their outcomes clearly.

_______________

1 For more information on the standards, see Paine Publishing, “Standards Central,” http://www.painepublishing.com/standards-central, accessed October 5, 2015.

Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×

MEASURES OF EFFECTIVENESS AND SUCCESS

Ms. Paine described six steps to achieving a quality of measurement that complies with the Barcelona Principals:

  1. Define your goals—Identify the role of communication in reaching the goal, defining the activity metrics (e.g., percentage increase in visits to AF.mil), and determining the outcome metrics (e.g., percentage growth in program funding or recruits) in terms of the mission.
  2. Understanding stakeholders—”Connect the dots” between stakeholders, prioritize them, and understand their need and interests.
  3. Define your benchmarks—Compare past performance over time or compare results with those of other services. The most important entity to measure against, Ms. Paine said, is “whatever keeps the generals up at night.”
  4. Define your metrics—Pick the right metric. The ideal index is actionable, is there when you need it, continuously improves your processes, and gets the results you need.
  5. Select data collection tools—Select the right tool(s) depending on what you are measuring.
  6. Use the data to make better decisions—Ask “So what?” three times, in order to understand what the data actually mean in real-world terms.

Ms. Wendi Strong asked how one might measure messages in advance (i.e., pre-launch testing). Ms. Paine recommended focus-group studies around the country for this purpose.

Col. Sean Monogue asked how one measures trust. Ms. Paine replied that it is mainly done with surveys. Turning the survey into a game is a useful technique, she said, because it fits with the gaming proclivities of today’s employees. It tests institutional knowledge in a way that makes the survey fun.

AVAILABLE ANALYTICAL TOOLS AND METHODS

A variety of specific tools and techniques used to measure communication effectiveness were presented and discussed. The appropriate tool depends on the objective:

  • Content analysis can be used to evaluate messaging, positioning, themes, or sentiment.
  • Survey research can be used to measure awareness, perception, relationships, or preference.
  • Web analytics can be used to measure engagement, action, or purchases.

As one example, Mr. Ed Brill said that IBM conducts a biennial “climate survey” of its employees. It finds considerable ambiguity in views of what IBM is. He said that, in contrast to the Air Force, lower-level employees tend to report that they have more information about IBM than managers do.

Tools also exist for mapping social networks and analyzing activities in social media. Ms. Paine described a method for measuring engagement with social media. Additionally, Foghound’s Ms. Lois Kelly had pointed out the need for a “predictive analytics capability” for measuring social media (see discussion in Chapter 3). Lastly, Mr. Robert Harles described technically sophisticated techniques for monitoring social media use and analyzing engagement, which together can form a “social media radar” for predictive capability.

KEY CONSIDERATIONS

In Mr. Harles’ view, setting measurable goals up front is critical—that is, how do you drive strategic impact? For example, he suggested focusing less on building large numbers of followers and more on building meaningful relationships with those who contribute valuable insight or influence others toward achieving stated objectives.

Ms. Paine said that it is essential not only to agree on metrics of success, as defined by senior leadership, but also to then tie rewards to them. In business, bonuses and salaries are tied to measurable outcomes.

Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×

A key role of employee feedback surveys, she continued, is to ascertain whether employees are able to connect their roles directly back to the corporate vision; the surveys should reveal whether the communication strategy is resulting in a culture of communication. Ms. Paine stated that value must be measured in real terms. In other words, data on reach, engagement, and impressions are less important than driving actionable insights, changing user behaviors, or influencing policy and strategic outcomes.

Several participants noted that the point of measuring communication is to have a basis on which to take action. For example, such data might allow the Air Force to determine the best way to deal with crises by overcoming negative or extraneous external communications. They can help determine what internal or external information is being used effectively and what is not. With these tools, the Air Force can use the data to inform important decisions. However, as Mr. Harles noted, “When you have all the analytics, you still have to synthesize that information”—or, as Ms. Paine put it, “Research without insight is just trivia.”

Theme 9—Measurement. Several measurement experts presented established tools and techniques for measuring the effectiveness of communication in an organization. As discussed, the process begins with senior leadership setting goals and determining desired outcomes—not simply outputs. Communication excellence can then be measured against quantifiable objectives and resulting performance, and the data can be used to improve decision making. Measuring social media use and analyzing how employees are communicating provides important awareness of the communication networks and key influencers within the organization. There are experts in industry who guide organizations in confidently and effectively planning and implementing a well-integrated communication strategy using the best available tools.

Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×
Page 23
Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×
Page 24
Suggested Citation:"4 Measuring Communication Effectiveness Across Diverse Backgrounds and Missions." National Academies of Sciences, Engineering, and Medicine. 2016. Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report. Washington, DC: The National Academies Press. doi: 10.17226/21876.
×
Page 25
Next: 5 Restatement of Main Themes »
Strategies to Enhance Air Force Communication with Internal and External Audiences: A Workshop Report Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Air Force (USAF) helps defend the United States and its interests by organizing, training, and equipping forces for operations in and through three distinct domains -- air, space, and cyberspace. The Air Force concisely expresses its vision as "Global Vigilance, Global Reach, and Global Power for America." Operations within each of these domains are dynamic, take place over large distances, occur over different operational timelines, and cannot be routinely seen or recorded, making it difficult for Airmen, national decision makers, and the American People to visualize and comprehend the full scope of Air Force operations. As a result, the Air Force faces increasing difficulty in succinctly and effectively communicating the complexity, dynamic range, and strategic importance of its mission to Airmen and to the American people.

To address this concern, the Chief of Staff of the USAF requested that the National Academies of Sciences, Engineering, and Medicine convene a workshop to explore options on how the Air Force can effectively communicate the strategic importance of the Service, its mission, and the role it plays in the defense of the United States. Participants worked to address the issues that a diverse workforce encompassing a myriad of backgrounds, education, and increasingly diverse current mission sets drives the requirement for a new communication strategy. The demographics of today's Air Force creates both a unique opportunity and a distinct challenge to Air Force leadership as it struggles to communicate its vision and strategy effectively across several micro-cultures within the organization and to the general public. This report summarizes the presentations and discussions from the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!