Click for next page ( 26


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
4 Using Data for Impact M ultiple challenges impede data-sharing efforts. Beyond differences in organizational culture and mission, simply establishing the processes that allow collection and sharing of data between orga- nizations can be costly and time consuming. To overcome these challenges, managers of data-sharing systems must consider how the data are to be used and how to get the most impact from the data. Only by improving the impact of shared data can better incentive be created for broad and sustained partici- pation in data sharing. In the third session of the workshop, three speakers provided examples of approaches to improve the impact of data sharing. DATA INTEGRATION AND VISUALIZATION New technologies have created the ability to gather, integrate, visual- ize, and disseminate data in ways that are qualitatively and quantitatively different from what has been possible before. Patrick Vinck, research sci- entist at the Harvard Humanitarian Initiative, described some of these new capabilities. New Software Thanks to new software, data collection and analysis are increasingly characterized by both precision and speed of acquisition. Advances in 25

OCR for page 25
26 DATA SHARING TO IMPROVE COORDINATION IN PEACEBUILDING software also enable rapid progression from data collection to analysis and dissemination, which can allow results to feed back into data collection. It is frequently possible to move from virtually no data to comprehensive data in a very short time. For example, an entire city can be mapped in just a couple of days using street mapping software and volunteers who are motivated to collect, assemble, and present information. One risk of new methods of data collection is that the amount of data collected can be overwhelming. Data therefore need to be aggregated and summarized. "I say `summarize' instead of `simplify,'" said Vinck, because data need to be made more consumable without decreasing their value. An especially useful way to summarize data is through the use of maps. For example, the LRA Crisis Tracker is a real-time data collection and map- ping platform that tracks the atrocities of the Lord's Resistance Army in Africa.1 Vinck also cited the Satellite Sentinel Project, in which the Harvard Humanitarian Initiative is involved, that seeks to deter atrocities by focusing world attention on threats to civilians.2 This project uses what a few years ago would have been military-grade satellite data for the purposes of protection and warning. Another project of the Harvard Humanitarian Initiative is PeacebuildingData.org, which seeks to give a voice to the people involved in peacebuilding and reconstruction processes. It features analyses and data from large-scale surveys in countries affected by mass violence and aims to bridge the gap between peacebuilding as intended by policymakers and its implementation and perception on the ground. Survey takers seek answers to questions such as: What have people experienced? How is the peacebuilding process affecting them? What do they think should be done? The informa- tion is collected digitally, which makes it faster to produce and results in better quality. Working in just a few countries, the project has sought to build a baseline of information that can be revisited every few years to gauge changes. It also can single out individual projects to determine whether they have been successful or not. An important application of such efforts is to help determine the extent to which the investments of the international com- munity have led to peacebuilding. 1 See http://lracrisistracker.com. 2 See www.satsentinel.org.

OCR for page 25
USING DATA FOR IMPACT 27 Data Platforms and Network Architectures Vinck emphasized the value of letting data speak for themselves rather than having researchers create a narrative. By presenting data through a technology platform, users can interact with the data. Vinck pointed to the PeacebuildingData.org project in Liberia (Figure 4-1), in which an online data set presented using Google Maps allows users to create indicators that are of interest to them. Similarly, for a project in Mindanao, Philippines, users can click on a list of indicators to access and visualize the information they want. So far, the information in the databases has come from a single source, but Vinck discussed the possibility of layering information from multiple sources onto a single map. Major questions that must be answered for such a system are whether information can be integrated and whether it is useful to do so. In part, he said, the answers depend on the purpose of the project. For example, a project focused on conflict analysis may differ from one focused on communication. Similarly, one project may lend itself to the development of a composite indicator that provides a peacebuilding score, while such an indicator might not be appropriate for a different project. Ideally, the data presented through interactive platforms would be com- pletely open to users. But data can be expensive and time consuming to col- lect, and letting go of data can be difficult. Data may also need to be protected FIGURE 4-1 Survey-Based Conflict Indicators for Liberia SOURCE: PeacebuildingData. org.

OCR for page 25
28 DATA SHARING TO IMPROVE COORDINATION IN PEACEBUILDING if they are from a sensitive source. PeacebuildingData.org has decided to let each organization display its own data rather than collecting the data in a centralized system. That way, each organization retains control over its data and can even choose to withdraw the data. Such an approach has implica- tions for updating data, Vinck acknowledged, as the information displayed can have different time frames and references. The development and use of metadata can help interactive platforms move toward networks of data sharing in which data have different sources but can be directly compared. Ethical Implications The collection, analysis, and sharing of data have important ethical implications, Vinck observed. For example, NGOs have various procedures for dealing with sensitive information, but those procedures typically are not shared with the public. How is information vetted? Are human subjects being protected? The lack of transparency makes it difficult to answer these questions for individual organizations. It also makes it difficult for organiza- tions to learn from each other. Widely disseminated guidelines, along with training on how to access and share information, can help organizations deal with issues that arise. ASSESSING VALUE IN DATA FOR DEVELOPMENT RESEARCH For data sharing to be effective, the data must have value. Innovations for Policy Action (IPA) is an NGO dedicated to demonstrating the value of data by discovering what works to help the world's poor. It designs and evaluates programs and provides hands-on assistance to bring successful programs to scale. It has more than 200 ongoing projects in about 40 (mostly developing) countries and offices in 14 countries. Niall Keleher, IPA's director of research methods and training, explained that the organization's long-term mission is not only to identify innovative social programs but to conduct multiple evaluations of programs in order to identify their impact in various contexts and with diverse populations. IPA begins by identifying not only the intervention to be used but the theory of change behind it. The organization then defines a representative sample for data collection, with particular attention to ensuring that the data accurately capture the populations about which statements are to be made. It does sample size and power calculations, applies valid randomization methods to the population, and develops indicators to accurately measure

OCR for page 25
USING DATA FOR IMPACT 29 the value of a program. Computer-assisted interviews provide access to data for prompt quality checks and offer the potential for more timely analysis. Distinguishing between causality and correlation is a challenge, Keleher acknowledged, but IPA's research design is carefully constructed to elicit "a true and unbiased estimate of the causal relationship between interventions and outcomes." The Approach in Practice Keleher described two examples from IPA's portfolio of projects. In northeastern India, the organization measured the success of an NGO seeking to achieve full immunization of children. It found that for an inter- vention organized around immunization camps where mothers brought their children for immunizations, the full immunization rate jumped from 6 percent for the control group to 18 percent for the group subject to the intervention. Furthermore, when the mothers received a one-kilogram bag of lentils as an incentive, the percentage jumped to 39 percent. "This kind of study is what we aim to produce--something that shows how effective a particular program was." A study in Malawi looked at the effect on repayment rates of tracking borrowers via fingerprint scanning technology. By having a photograph of a person and a fingerprint in a database, the highest-risk borrowers sub- stantially increased their repayment rates, enabling others to obtain loans whereas before they might have been denied because of risk. Access to some of the data collected by IPA is limited by confidentiality and intellectual property considerations. But in general IPA seeks to make its data available through publication in scientific journals. This transparency encourages others to try to replicate the evaluation technique, validate the published data, and build on previous results. UNDERSTANDING FRAMES OF REFERENCE Stephen Lowe, geospatial information officer in the Office of the Chief Information Officer at the US Department of Agriculture, discussed some of the issues that arise in interagency data sharing. First, he said, factors mentioned in the discussion of data sharing and peacebuilding occur across the federal government. Many government agencies and personnel have different frames of reference--agendas and ways of communicating--yet they face common issues involving data. Where should data sharing start

OCR for page 25
30 DATA SHARING TO IMPROVE COORDINATION IN PEACEBUILDING and stop? What are the scope and scale of data sharing needed for a given project? When should data sharing focus on interpretation, and when on discovery? Are the facts available but extremely complex, or are missing facts creating uncertainty? (As Lowe said, "Sometimes we don't know what we don't know.") What is the appropriate tradeoff between data precision and speed of acquisition? Some data are more valuable when they are acquired quickly, as opposed to gathering more precise data over a longer time frame. Lowe also mentioned a more fundamental difficulty with data shar- ing: the distinction between policy disagreements and policy controversies. Policy disagreements involve disputes in which the two parties are able to resolve the questions at the heart of the dispute by examining the facts of the situation. Policy controversies are disputes that are immune to resolu- tion by appeal to facts, making them much more intractable. Furthermore, people can focus their attention on different facts or interpret the same facts in different ways, and they have a remarkable ability, when embroiled in a controversy, to dismiss the evidence cited by their antagonists. Seeing Outside the Frame People tend to interpret evidence based on the frames of reference they apply, Lowe explained. These frames incorporate beliefs, perceptions, and appreciations that underlie policy positions. They have normative implica- tions that a certain type of solution is acceptable. To overcome barriers created by different frames of reference, people need to seek agreement on the nature of the problem and the general char- acter and content of a solution, said Lowe. The type of problem to be solved may involve diagnosis, classification, analysis, the detection of anomalies, the configuration or selection of data, monitoring, prediction, design, or planning. Understanding the type of problem leads to better alignment with different types of available solutions. Lowe further explained that understanding the framing of a problem can create opportunities to operationalize solutions. Some problems lend themselves to customized one-of-a-kind solutions, while others may yield to highly standardized and routine solutions. By moving toward the lat- ter, unit costs can be reduced and efficiencies realized. Understanding the framing correctly can enable the proper use of technology within the data acquisition workflow. For example, in certain contexts, data can be collected automatically by a sensor detecting activity in realtime in its vicinity or it

OCR for page 25
USING DATA FOR IMPACT 31 could be collected manually using a template available on a tablet computer. The framing helps identify the best mobility solution for data acquisition. Lowe concluded by observing that maps are typically created for people who are already in power, who control the resources to create these docu- ments. New technologies may make it possible to flip that equation around. For example, community mapping using volunteers can empower people in communities by identifying emergent issues, grounding conversations in context, and depicting local knowledge and values. DISCUSSION Kevin Brownawell, interagency professional in residence, US Institute of Peace (USIP), cited some of the difficulties that can arise in using new technological capabilities for data sharing. First, some of the most useful data that can be collected are subjective and designing survey instruments to collect this kind of information can be much more labor intensive than col- lecting objective data such as immunization rates. Similarly, data input can be very labor intensive. A large and well-trained staff is generally required to input large volumes of data, regardless of technological capabilities. At the same time, the quality of the data needs to be assessed, which requires an investment of time from well-trained personnel. Finally, interpretation of the data can be difficult and contentious. "Who is going to interpret the data? What type of framework do they have?" At USAID, he said, he and his colleagues often worried about passing controversial data up the chain of command, because senior officials had a tendency to interpret the data in ways that reflected their circumstances rather than the context in which the data were gathered. Keleher agreed that subjective data are often the only data that can be collected given the focus of his organization's work. However, these qualita- tive data can help interpret more quantitative measures. While he agreed that methods of collecting subjective data can be methodologically rigorous, he observed that new technologies can ease data-gathering demands. For example, when a delivery man makes a delivery, it is recorded using a simple hand-held device, and the information thus collected can be valuable in an organization's decision making. Thus, a major component of an organiza- tion's planning should be careful decisions about what are the important data to collect and how to collect those data.

OCR for page 25
32 DATA SHARING TO IMPROVE COORDINATION IN PEACEBUILDING In response to a question about data reliability, Vinck talked about some challenges in crowdsourcing. For example, in some parts of Liberia, cell phone ownership is much lower than in other parts. "In terms of reporting, that has a major impact," he said. In such situations, there are substantial advantages to having trained people gather data, despite the greater effort required for training and sending them into the field. He also advocated that peacebuilders learn more about how to maintain the quality of data collection and analysis. To check the data they collect, researchers can triangulate information from different sources. The tech- nologies used to collect data also make it possible to check consistency and the reliability of interviewers and the information they gather. Research protocols have strict standards concerning how to select interviews, how to conduct them, and how to get consent. Peacebuilders also need to under- stand research design and the problems with flawed research approaches. "Training needs to be done on how to use and access data and also how to judge and understand data." Richard Boly, the director of e-diplomacy at the State Department, agreed that a centralized database under the control of a single entity is not feasible, and added that citizen-generated data can both validate data gener- ated by the government and result in data generated independently from the government. This open model of data sharing can support not only decen- tralized data gathering but also decentralized analysis. Lowe emphasized the importance of the metadata description of an information asset so that it is searchable and accessible from a variety of interpretive stances. Good metadata allow data to have a much longer life cycle and greater usefulness. He also cited the importance of multiple inter- pretations of data, which require that the data be capable of being pulled apart and being used in a different way. This may not mesh well with current business models, but new technologies allow this kind of open-source data gathering and analysis, which create great promise for the future. Susanna Campbell, research fellow at the Saltzman Institute for War and Peace Studies, Columbia University, noted that the uses to which data are put may constrain the selection of data collected and analyzed. She pointed to four kinds of uses of data for peacebuilding. The first is to improve the effectiveness of programming, generally through the monitoring of ongoing programs. These data are generally not shared, because people are less likely to provide full assessments if they know that what they say will be freely available. The second is to categorize peacebuilding successes and failures. These data are more likely to be shared because they are more likely to be

OCR for page 25
USING DATA FOR IMPACT 33 part of an academic study than an effort to improve programming. The third is to improve coordination, which often relies on open data from the community. The fourth is to improve the targeting of programming, which requires data on the context in which programming occurs. These data can be particularly useful in demonstrating the interrelationships among systems and how systems work together. Anne Ralte, senior advisor in the Office of the Director of Human Resources for USAID, mentioned the Standardized Monitoring and Assess- ment of Relief and Transitions (SMART) system, a USAID initiative.3 As indicators for humanitarian systems, the system uses the overall mortality rate, which is a crude and somewhat controversial indicator, and the nutri- tional status of children under age six. Many organizations have bought into the effort, and the data are now housed in the Center for Research on the Epidemiology of Disasters in Brussels. Data are gathered by NGOs and vali- dated by a group of independent epidemiologists, with a simple-to-use and standardized data-gathering tool. Graphical presentations of the data have been developed to improve interpretation and dissemination. Vinck mentioned another project called Food and Nutrition Technical Assistance (FANTA), which uses standardized indicators of food security and nutrition.4 He noted that even something as straightforward as perceptions of security measured every three years can be a very useful measure, provid- ing a baseline against which to gauge progress (or lack of it). Hartwell observed that maps can be highly incendiary--for example, by drawing attention to disputed boundaries. Chip Hauss, director of the Alli- ance for Peacebuilding, responded that maps in and of themselves are not dangerous: it is how they are used that can create disruption. Maps can be very efficient and effective tools, but, like statistics, maps can also lie. Andrew Robertson, senior program officer at USIP, pointed out that in the workshop's morning discussions, trust was described as coming from dialogue, whereas in the current session, trust comes from method and struc- ture. Method and structure point to the need for planning and the ability to predict relevant questions, but adaptability, flexibility, and learning are also crucial to successful peacebuilding. This shift is happening in the commercial world, from a structured to a more flexible and adaptive approach. Tools therefore need to be quickly adaptable to adapt to changes in what stakehold- ers think they need. This is where the morning and afternoon discussions could fit together, he said. 3 More information about the system is available at www.smartindicators.org/index.html. 4 For more information, see www.fantaproject.org.

OCR for page 25
34 DATA SHARING TO IMPROVE COORDINATION IN PEACEBUILDING Andrew Blum, director of learning and evaluation at USIP, observed that even domestic data-gathering projects may have lessons for peacebuilding. For example, efforts to gain information on immigrants or families with at- risk children have many commonalities with peacebuilding data-gathering efforts. Collaborations among organizations doing different kinds of data sharing and dissemination would enable the sharing of lessons learned and best practices. At the end of the session, Roman stressed the importance of incentives for different parties to share information. New capabilities depend critically on improving the information flow among .org, .gov, and .com information domains. Thus, collaboration in a decentralized framework will be essential to the creation of data-sharing mechanisms for peacebuilding.