4
Measuring Communication Effectiveness Across Diverse Backgrounds and Missions
As with any program, measuring the effectiveness of communication is vital to understanding its value and learning how to shape it for greater effectiveness.
Brig. Gen. Kathleen Cook expressed to the committee a desire for definable, measurable, results-based outcomes, using the approach that “we are trying to change the knowledge, attitudes, or beliefs of a specific audience to achieve a specific action.” This clarity, she said, would not only inform the message, but also provide a construct under which to align the U.S. Air Force’s (USAF’s) weight of effort and inform what they should be measuring themselves against.
DEFINING “EFFECTIVE COMMUNICATION” IN THE AIR FORCE CONTEXT
What is “effective” communication? Akin to weapons deployment, it can be described as communication that reaches its target, impacts its audience, and achieves the intended objective for that particular audience. Several participants stated that whether in the context of an individual message or an Air Force-wide communication strategy, this requires setting objectives, planning, and acting in a deliberate and organized fashion.
The discussion among several of the workshop participants highlighted that in the process of planning the elements of a communication strategy, leadership should first define what they hope to achieve. These participants acknowledged that communication requires measurement, and the selected measures must be specific, measurable, attainable, relevant, and timely. To that end, they noted, leadership needs a clear and consistent vision of what success looks like; this vision of success should be tied to goals.
Ms. Katie Paine, chief executive officer of Paine Publishing, established the context for discussing this topic at the workshop by summarizing the Barcelona Principles, a set of industry standards for measuring communications effectiveness and outcomes, developed by an assembly of communication-measurement professionals in Spain.1 A few workshop participants discussed that an Air Force communication “baseline” could be defined as instituting within the communication culture a sense that the Air Force must measure its communications in order to understand their outcomes clearly.
_______________
1 For more information on the standards, see Paine Publishing, “Standards Central,” http://www.painepublishing.com/standards-central, accessed October 5, 2015.
MEASURES OF EFFECTIVENESS AND SUCCESS
Ms. Paine described six steps to achieving a quality of measurement that complies with the Barcelona Principals:
- Define your goals—Identify the role of communication in reaching the goal, defining the activity metrics (e.g., percentage increase in visits to AF.mil), and determining the outcome metrics (e.g., percentage growth in program funding or recruits) in terms of the mission.
- Understanding stakeholders—”Connect the dots” between stakeholders, prioritize them, and understand their need and interests.
- Define your benchmarks—Compare past performance over time or compare results with those of other services. The most important entity to measure against, Ms. Paine said, is “whatever keeps the generals up at night.”
- Define your metrics—Pick the right metric. The ideal index is actionable, is there when you need it, continuously improves your processes, and gets the results you need.
- Select data collection tools—Select the right tool(s) depending on what you are measuring.
- Use the data to make better decisions—Ask “So what?” three times, in order to understand what the data actually mean in real-world terms.
Ms. Wendi Strong asked how one might measure messages in advance (i.e., pre-launch testing). Ms. Paine recommended focus-group studies around the country for this purpose.
Col. Sean Monogue asked how one measures trust. Ms. Paine replied that it is mainly done with surveys. Turning the survey into a game is a useful technique, she said, because it fits with the gaming proclivities of today’s employees. It tests institutional knowledge in a way that makes the survey fun.
AVAILABLE ANALYTICAL TOOLS AND METHODS
A variety of specific tools and techniques used to measure communication effectiveness were presented and discussed. The appropriate tool depends on the objective:
- Content analysis can be used to evaluate messaging, positioning, themes, or sentiment.
- Survey research can be used to measure awareness, perception, relationships, or preference.
- Web analytics can be used to measure engagement, action, or purchases.
As one example, Mr. Ed Brill said that IBM conducts a biennial “climate survey” of its employees. It finds considerable ambiguity in views of what IBM is. He said that, in contrast to the Air Force, lower-level employees tend to report that they have more information about IBM than managers do.
Tools also exist for mapping social networks and analyzing activities in social media. Ms. Paine described a method for measuring engagement with social media. Additionally, Foghound’s Ms. Lois Kelly had pointed out the need for a “predictive analytics capability” for measuring social media (see discussion in Chapter 3). Lastly, Mr. Robert Harles described technically sophisticated techniques for monitoring social media use and analyzing engagement, which together can form a “social media radar” for predictive capability.
In Mr. Harles’ view, setting measurable goals up front is critical—that is, how do you drive strategic impact? For example, he suggested focusing less on building large numbers of followers and more on building meaningful relationships with those who contribute valuable insight or influence others toward achieving stated objectives.
Ms. Paine said that it is essential not only to agree on metrics of success, as defined by senior leadership, but also to then tie rewards to them. In business, bonuses and salaries are tied to measurable outcomes.
A key role of employee feedback surveys, she continued, is to ascertain whether employees are able to connect their roles directly back to the corporate vision; the surveys should reveal whether the communication strategy is resulting in a culture of communication. Ms. Paine stated that value must be measured in real terms. In other words, data on reach, engagement, and impressions are less important than driving actionable insights, changing user behaviors, or influencing policy and strategic outcomes.
Several participants noted that the point of measuring communication is to have a basis on which to take action. For example, such data might allow the Air Force to determine the best way to deal with crises by overcoming negative or extraneous external communications. They can help determine what internal or external information is being used effectively and what is not. With these tools, the Air Force can use the data to inform important decisions. However, as Mr. Harles noted, “When you have all the analytics, you still have to synthesize that information”—or, as Ms. Paine put it, “Research without insight is just trivia.”
Theme 9—Measurement. Several measurement experts presented established tools and techniques for measuring the effectiveness of communication in an organization. As discussed, the process begins with senior leadership setting goals and determining desired outcomes—not simply outputs. Communication excellence can then be measured against quantifiable objectives and resulting performance, and the data can be used to improve decision making. Measuring social media use and analyzing how employees are communicating provides important awareness of the communication networks and key influencers within the organization. There are experts in industry who guide organizations in confidently and effectively planning and implementing a well-integrated communication strategy using the best available tools.
