8

Closing Remarks

The final session of the workshop provided closing remarks by members of the steering committee, followed by open discussion with the audience. Panelists were asked to identify their ideas for next steps to improve health research for small populations. Gordon Willis (National Cancer Institute [NCI]) moderated the session.

STATEMENT OF GRAHAM COLDITZ

Colditz reflected on the value of federal initiatives to add extra data classifiers, especially those described by Howard Koh (see Chapter 2). He asked about other data classifiers. If needed, he suggested, a goal could be to find the leadership to design and implement classification information at the federal and state levels. As an example, at a workshop held by NCI in fall 2017, one session described some of the many different federal definitions of “rural.” Consistency in definitions is important to move research forward at the fastest possible rate.

He observed that the valuable work by Scarlett Lin Gomez and others using relatively new classifiers for Asian subgroups and lesbian, gay, bisexual, and transgender (LGBT) communities raises the question of which subpopulations have not been studied. Electronic health records are a great potential source of information for many, but will not include people who do not access health care. As an example, he cited the Mississippi Delta. Researchers try to study these populations, but it takes fieldwork. Not everyone in the United States has a health record. He observed even NCI

cancer control catchment areas may miss the small underserved rural populations who live outside the boundaries of the 80 percent catchment areas.

STATEMENT OF JAMES ALLEN

Allen focused on efficiency, optimization, and generalizability, noting their particular relevance to research with small populations. Allen also noted convergence in small population research with a number of ethical considerations; small population research is critical to efforts that seek to address health inequity, and its advancement has numerous social justice implications.

There has been an explosion of new analytic approaches that afford unparalleled opportunities to ask and provide answers to an array of new questions by exploring complex interactions and multivariate causation. Allen noted that these developments, with the possible exception of Bayesian methods, have happened without accompanying consideration regarding efficiency in their use of information or optimization strategies that foster efficiency. As a result, these approaches generally require large sample sizes, usually precluding application in small population research that is typically in the position of drawing inferences from a limited number of observations.

This lack of efficiency imposes limitations when researchers seek to study health equity by inhibiting the research possible with small populations, some of whom are experiencing the largest existing health inequities in the United States. It becomes a social justice issue in that it privileges people from certain populations (those that can produce large samples and are easily assessible to researchers) in the research over others. Allen concluded that as a result, current methods have often led to a lack of empirical data to guide development of interventions and establish what is effective for the small populations that are often facing the largest health inequities. Optimization strategies for contemporary statistical and measurement approaches and alternative research designs that make more complete use of available information—along with mixed methods that provide new possibilities in qualitative research and Bayesian approaches that optimize through use of prior information—provide rigorous approaches to study important research questions that can only be addressed through small population research. Allen noted that the promise of many of the new approaches described at the workshop is in how they permit scientists to address significant research questions, instead of abandoning the questions because they cannot be investigated.

A third emergent theme in small population research is the importance of understanding context, highlighting the role of distinctiveness in such areas as culture, geography, setting, and history in understanding health

outcomes. This element in small population research may be at odds with methods that attempt to control for distinctiveness in order to explore and test generalizable conclusions. Yet, local distinctiveness is central to the small population paradigm.

Allen noted that generalizability is a controversial issue among many small population researchers because of its implications regarding what research gets funded. Research findings that can generalize across groups are prioritized. The problem is that this discourages research that studies the distinctiveness that is often at the heart of what is responsible for the health inequity in a small population. Allen emphasized that though research with small populations can often be generalizable to other groups, broad generalizability should not always be a mandatory requirement. Findings are often needed on what might be uniquely relevant to a particular health problem in the group or to interventions that work for a particular group. When a researcher arrives at a finding not immediately generalizable across populations, it potentially can represent an important adaptation to a problem within that particular context, or an element of a health intervention that, in working for that particular group, explains why existing interventions have not been successful in the setting. From the perspective of small populations, it may well be that such nongeneralizable findings have value on their own.

In closing, Allen cited a quote from a letter from Vincent Van Gogh to his brother Theo, in which he wrote, “Great things are done by a series of small things brought together.” Allen noted this is also a good summary of the workshop discussions over the past 2 days.

STATEMENT OF GRAHAM KALTON

Kalton noted the challenge in justifying a study of a small population to federal funders. One approach is to document population differences with respect to an outcome variable. However, in many cases, the dilemma of “no data, no problem” makes this difficult to document. He said proposals should argue the case clearly about why the target population is important to study.

In response to Allen’s comment on generalizability, Kalton said he finds it hard to see how a tribe or area should not scientifically be thought of as part of a broader population, though defining what that broader population is may be a challenge. He asked whether research on a 200-person tribe, with no generalizability beyond the tribe, would warrant a study. He suggested one way to achieve this is through replication. In other words, a researcher would not necessarily study just one tribe but team up to study other tribes as well. The purpose would be to start to understand whether

or not there is variation. He reiterated the idea of reproducibility as an important concept that needs more attention.

Kalton referred to the workshop discussions about how to define small populations. The LGBT population, for example, is very diverse depending on how it is defined. The homeless are similarly poorly defined. He observed that makes it difficult to think of replication, although replication is needed to build up the knowledge base.

He described similar definitional issues with disability studies, saying that surveys measuring disability in the past gave very different estimates in part because they used different definitions. However, a federal initiative among statistical agencies developed a question to measure disability, which he called a step in the right direction.

Related to electronic health records and registries, Kalton argued for research to determine the quality of these data. He cited a quote from the Josiah Stamp (1929):¹ “The government are very keen to amass statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman who just puts down what he damn pleases.” Kalton summarized steps that have moved the survey world forward since that time, such as standardization and training. But with electronic health records, the clinicians who are entering data are not necessarily concerned about research data needs. The health record part may be fine, but some of the classifications may be based on judgment. Researchers need to understand the quality of the data. Similar challenges exist with registries. He has found, for example, that most people who are Hispanic are not recorded as Hispanic in Medicare records. With any new data source, the researcher needs to understand all the error components. He noted that survey researchers have adopted the concept of total survey error and suggested that the same kind of thinking needs to be applied to the emerging data sources used for health research.

Kalton noted that for samples produced by some methods for sampling very rare and hidden populations described in Chapter 4, representativeness depends on a number of questionable assumptions. Even though these methods are problematic, there is often no alternative solution. Researchers need to acknowledge the limitations of these methods and assess the possible effects on their findings. We need to think about how to check the quality of the samples obtained, he suggested. He commented on the use of technology in the survey field, stating it will be very powerful for the future. Researchers should be looking at the uses of Apple Watches, Fitbits, smartphones, and other new technologies. He noted that technology has two potential benefits.

___________________

¹ Stamp, J. (1929). Some Economic Factors in Modern Life, pp. 258-259. P.S. King & Son, Limited. Original from the University of Michigan.

First, it may improve the quality of the information collected, providing more accurate, or otherwise unattainable, measurements at any one time, as well as giving measurements over time. Second, these technologies may sometimes make it less burdensome to participate. Respondents may find these novel technologies to be more attractive modes of data collection. These attributes may help to counteract to some degree the decline in response rates that surveys are currently experiencing.

STATEMENT OF JANICE PROBST

Probst focused her remarks on ethics and the importance of community. She noted that her research focuses on rural vulnerable populations, specifically rural minority populations. Rural, a geographic measure, and race/ethnicity, a self-reported social variable, fall into the category demographic characteristics. In most government reports, and in most surveys documented in the literature, demographic characteristics are presented individually: for example, a line for age, a line for gender, a line for race, and maybe a line for residence. The intersection between characteristics is less often presented. But “crossing” characteristics—for example, looking for interactions between race/ethnicity and geography—brings to light important subcategories, such as the large rural minority population and its health disparities. Dual disparities can be missed when investigators use race as one term in a model, and residence as another, and do not examine the interaction. Potentially there are additional intersections that are never examined, such as poor white people in the inner city.

In closing, Probst suggested researchers think of the ethics behind their assumptions and approaches as they deal with racial groups with different characteristics. In addition, they should consider the potentially hidden interactions between residence and other personal characteristics.

STATEMENT OF LANCE WALLER

Waller said he often discusses with students that interdisciplinary work is fun and challenging. The interdisciplinary nature of this workshop showed that researchers have been looking at common problems and thinking about them in different ways.

He suggested researchers should consider the big question they want to answer, such as health impact, description of disparities, or reasons for disparities. They should think about the data, measures, and methods that would that would help them answer the question exactly right, practicalities aside, then ask what kind of data they can get with the methods available. The goal of this cycle is to see how close the question that can be answered is to the original question. In every cycle, the goal is to learn more

than last time. He noted that the presentations by Patrick Sullivan, Vetta Sanders Thompson, and Kathi Mooney highlighted different types of data collection. None of them claimed to have the answer, and they continually conducted evaluations to assess how well something worked. The idea of continuously evaluating approaches is often overlooked.

A potential bias may exist if social media are used to contact people, but the researcher should evaluate whether the bias matters for the current study, he suggested. Routinely testing the coverage and the response rates during a study is a very helpful idea.

Many designed experiments are set up for very controlled situations. However, small population studies may involve uncontrolled situations. That does not mean design is irrelevant, but it has to be developed. An ongoing assessment approach would facilitate learning.

Waller noted that new technology and privacy and confidentiality issues are interesting. He was involved with the study Sullivan described about the sex-seeking app (see Chapter 4). A graduate student walked around town with his smartphone and soaked up data that people had put out in public. People advertise themselves for one purpose, and researchers are using the information for another. This is just one of many issues that will continue to arise.

He related he was part of a panel discussion about nonbinary assignment of gender in public health studies. Students said they are tired of having male and female be the example of zero-one variables. He made the point that good epidemiologists or statisticians will find this a challenging and interesting problem to work on, rather than something impossible. The question of classifiers will be an ongoing issue to be addressed.

OPEN DISCUSSION

Mandi Pratt-Chapman asked the statisticians whether they had considered using qualitative comparative analysis as an alternative to some of the statistical approaches discussed at this workshop. She explained that qualitative comparative analysis comes out of the sociopolitical literature and is based on Boolean logic. She suggested the workshop discussions seemed to prioritize quantitative methods and how to get things to fit within the assumptions or parameters of current methods, rather than explore the potential of additional approaches. She said she likes the idea of using mixed methods analysis (both quantitative and qualitative) and including input from the community. In her view, if a study done with a tribe of 200 individuals solved a problem leading to wellness within that community, it was successful whether it could be replicated or not. Waller responded the problem may be a training gap. He said he has learned from colleagues in the behavioral sciences, but does not have their same internalization of

framing questions, for example. Similarly, his colleagues in the behavioral sciences tend to have little training in quantitative methods. As a result, there is not enough common ground to have a good discussion about the value of each approach. Another issue is having the humility to admit that a knowledge gap is important to achieving rich collaboration.

Kalton agreed with the value of qualitative work. Quantitative work is rigid but has the benefit of leading to generalizability. The qualitative approach enables researchers to explore new things and learn different dimensions. He noted that the presentation by Scutchfield about Kentucky’s approach to improving health in rural areas used extensive collaboration at all levels. Scutchfield provided a convincing case that their approach and the procedures they have put in place have benefited the rural population in Kentucky. He also was interested in the discussion of Asian Americans and the fact that together they make up a meaningless grouping for some purposes. The grouping needs to be subdivided in order to create meaningful subgroups for some analyses. The challenge is, how do we know that in advance?

Colditz noted implementation science is pushing for interdisciplinary bridging, an approach that is important to furthering research on small populations as well.

Chris Fowler commented that the way research is conducted may hide subgroups. He urged making use of geography. Why people live where they do is a complex outcome of many processes. The intersection of all those processes produces an outcome that often has distinct spatial patterns to it. Looking at those patterns may lead to identification of subgroups and subpopulations. An opening up in this one area of information may provide leverage into other areas, Fowler said.

Tom Louis noted the presentations illustrated creative and innovative ways of contacting, recruiting, and retaining participants. He asked whether more might be learned if small randomized experiments were embedded in a project to learn something generalizable. He encouraged colleagues to look for a chance to randomize and do little experiments.

Kelly Devers made four points. First, related to engagement and trust, she advocated for continuing to engage small and hidden populations, hear their voices, and understand what they do or do not want. Second, mixed methods work with small population groups and can help improve everything from recruitment to new ways of analyzing the data. She added that qualitative comparative assessment is also called meta-analysis and should be considered as a tool in improving health research for small populations. Third, a number of presentations urged leveraging new technology and new data, but it is necessary to validate those data, understand what they tell and do not tell, and assess some of their strengths and limitations. Finally, she said implementation science is trying to move findings into practice

in a variety of settings. She suggested that at the beginning of a study, the researcher needs to think about what is practical to do and how quickly. Implementation science experts should be involved at the beginning of projects to help get useful ideas into practice as soon as possible.

Jessica Xavier said the question of definitions is valid and important. More time is needed to develop methodologies that accurately capture those populations to ensure the quality of the information. She responded to Waller’s comment on gender. Half of the transgender youth who are coming out are gender nonconforming. They have a myriad of different identities, sex practices, ways of relating to health, and health care needs. She asked how the government can fund research and interventions to serve this huge emerging population. These emerging identities will have meaningful implications for how to serve these populations, guaranteeing them health care access in the future.

Rick Moser asked how NCI should decide which small population or subpopulation to be studied. He stressed that “no data” does not mean “no problem.” Waller suggested a balance and collaboration between funding agencies and investigators. Funding agencies should look for the best work being done by the people who have the best handle on the question posed. That said, it is hard to prioritize the small studies, referring to many examples presented during the workshop. He said it is important to build on what is known, try to improve, and move the question researchers want to answer and the question they can answer closer together.

In response to Moser, Rina Das from one of the cosponsoring agencies, the National Institute on Minority Health and Health Disparities, said that her institute has the privilege of making decisions as to what underserved populations are critical to study. She said that they have a definition of health disparity populations that initially included racial/ethnic minorities, low socioeconomic status, and underserved rural people. They then promoted and now include sexual gender minorities as a health disparity population. Within federal agencies, her institute and other agencies pay attention to which underserved populations are evolving and where there is a need to conduct more research. She reminded the audience about many NIH research and funding opportunities in these population areas.

Pratt-Chapman posited, “Change happens at the pace of caring.” She praised the workshop, but warned against an us/them dynamic between researchers and small populations. She said the question about which populations to study is not going to be answered until the research workforce reflects the diversity of the small populations.

Shobha Srinivasan provided final comments. She thanked the organizers and participants, noting the workshop showed the need to carry on, try, and perhaps adapt various methodologies, designs, statistics, and qualitative methods. Many of the presenters are developing methodologies as they

are going on with their work, she observed. She described the two populations that started her thinking about the issue of small populations. One was a challenge about 10 years ago, trying to respond to questions about the lack of research on the American Indian and Alaskan Native communities. She said as she talked to various tribes, she realized researchers were insensitive to the fact that the tribes are sovereign nations and had real questions. The other group that she found had been completely disregarded from the studies within her division was the rural population, explaining that it is a sample size and accessibility issue. She said doing this research is a question of morality and ethics.

Srinivasan thanked the partners who helped convene the meeting, the steering committee, and the National Academies staff.

Steering committee chair Colditz thanked the participants and suggested they respond to opportunities to refine understanding and improve the health of small populations.

This page intentionally left blank.