8

Evaluating Science Communication

Well-designed and rigorous evaluations can help answer critical questions in science communication. What types of narratives and images produce accurate impressions and lasting memories? How does a presentation’s impact vary by audience? How should statistics and human narratives be combined for maximum impact? As the narrator of the session on evaluating science communication, Arthur Lupia, Hal R. Varian Collegiate Professor of Political Science at the University of Michigan, put it, “many researchers and science organizations have limited resources to devote to communication. Many of you are being asked to do more with

less. How can we create effective science communication strategies in these circumstances?”

INFLUENCE ON TWITTER

James Fowler, professor in the Political Science Department and in the Global Public Health Division of the Department of Medicine at the University of California, San Diego, addressed the evaluation of a specific kind of communication: tweets that involve science. Many measures exist of the influence of scientists on social media, Fowler said, including levels of activity, the responses generated by a person’s activity, and a person’s influence within a network of people (Riquelme and González-Cantergiani, 2016). However, none of these measures necessarily mean that a person is influential. “You could be saying things that are completely banal to millions of people and not changing their lives at all,” said Fowler. “That’s where a lot of these measures fall down, because they’re just measuring quantity.”

On Twitter, measures of quantity specifically include tweets, replies, retweets, favorites, mentions, follows, and network indicators. However, these measures have the same problem as with other social media: they do not measure influence. “It reminds me of an ancient university proverb: ‘Academic deans can’t read, but they sure can count,’” said Fowler. “The literature has been focusing more on what a dean would focus on.”

Fowler described a way of measuring influence that incorporates quality as well as quantity. The method begins with the measure of information in a tweet. For example, if a girl gets socks from her aunt every Christmas, opening that year’s present and finding socks does not provide much information. But if the present is a brand new bicycle, that girl’s ideas about her aunt will change, because her aunt has done something that she has never done before.

Similarly, the quality of a tweet can be quantified by measuring how likely it is that a tweet would have occurred. According to information theory, the less probable something is, the more information it contains. Specifically, the negative logarithm of the probability can be summed across different pieces of information to get the total information in a message.

To measure the probability of a tweet, Fowler took the simple and transparent method of looking at all the words on Twitter that have occurred in the past 24 hours. He then measured all the words among responders for a 24-hour period and classified how likely it was that a given tweet would have randomly drawn from the set of all the words on

Twitter. “This is a good place to start because there will be some information in a tweet based on the kinds of words that are used” in responses to that tweet.

SCIENTISTS ON TWITTER

Scientists tend to be in the middle of the distributions of user favorites, followers, and friends among all Twitter users, Fowler noted. However, they tend not to tweet as much. “We’re busy, and tweeting isn’t necessarily the first thing we do.” As a result, scientists get fewer retweets per person, fewer replies per person, and fewer quotes per person. “That’s not surprising. We have lower activity, so there’s going to be fewer responses,” Fowler said. They also get less information into the Twittersphere, because they tweet less.

However, the tweets of scientists have higher average information scores than do the tweets of nonscientists. “The moral of the story is that we don’t tweet as much as other people, but when we do tweet, we tweet with a higher amount of information,” he noted.

This higher information content can result in great impact, Fowler observed. As the information score of a tweet goes up, the likelihood of retweets, replies, and quotes increases. Furthermore, this increased interest can have concrete outcomes. A study of Facebook users, for example, found that messages on Facebook led about one-third of 1 million more people to vote in 2010 than would have been the case otherwise (Bond et al., 2012).

This is the kind of approach that is needed to move beyond quantity to quality, Fowler concluded. Researchers need to “try to think of creative ways where, at scale, we can measure the quality of [messages]. It’s only then that we’re going to be able start to think about how one scientist’s actions might be able to change other people’s opinions.” For instance, could messages from scientists change policy or encourage people to become involved in citizen science? “I feel optimistic, given the comparisons that we’re now just starting to make.”

In addition, measures of quality may encourage scientists to communicate more and build more of a following, further increasing their influence. Scientists “are late comers to social media,” said Fowler, “but now a lot of them have adopted it.”

This page intentionally left blank.