The first session of the workshop explored pain points and opportunities for using digital health technologies (DHTs) for characterizing disease in pre-clinical research. The speakers offered a range of different perspectives, including a nonprofit platform for research, a government-run national data collection project, a patient with lived experience collecting and navigating health data, and a platform for enabling participatory discovery using person-generated health data (PGHD). Larsson Omberg, vice president of systems biology at Sage Bionetworks, discussed how incorporating DHTs into research protocols can capture real-world data that can be used to glean health insights that may be missed by traditional clinical measurements. He also described challenges encountered in collecting data “in the wild,” such as confounding factors that can affect the interpretation of the data. Chris Lunt, chief technology officer at the National Institutes of Health’s (NIH’s) All of Us Research Program1 (All of Us), discussed experiences and lessons learned from the program’s efforts to collect digital data on a large scale. Lunt described the value of data collected with DHTs and shared the program’s criteria for prioritizing their
___________________
1 The All of Us Research Program was created in 2015 by the National Institutes of Health with the aim of collecting data from 1 million volunteers to accelerate health research and improve the provision of individualized care. Information about the project is available at https://allofus.nih.gov (accessed May 9, 2020).
assessment and device strategies. Alicia Staley, senior director of patient engagement at Medidata Solutions, offered a patient’s perspective on the distinction between clinical care and clinical research. She suggested that engaging with patients and more effectively taking into account the consumer mindset could help drive more widespread adoption of DHTs in the research and clinical care domains. Luca Foschini, chief data scientist and co-founder of Evidation Health, described how PGHD can enable participatory approaches to the discovery and development of interventions. The session was moderated by Effy Vayena, director of the Health Ethics and Policy Lab at ETH Zurich.
CHALLENGES IN DERIVING HEALTH INSIGHTS FROM REAL-WORLD SENSOR DATA
Larsson Omberg, Vice President of Systems Biology, Sage Bionetworks
To explore the challenges and opportunities associated with the use of real-world sensor data to derive health insights, Omberg drew on experiences and lessons learned from conducting research using data collected from mobile phones and wearable devices. Incorporating DHTs into research protocols enables the collection of data in a real-world setting—or “in the wild”—and can increase the volume and diversity of the data, Omberg explained. In traditional clinical research protocols, measurements are typically collected intermittently when patients come into a clinic, with large time lapses between visits. Research protocols that integrate DHTs allow for more frequent or even continual assessment using sensor measurements and the ability to collect data that captures the interaction between a person’s health and their environment. In this type of protocol, the interaction between the research protocol and the participant’s life drives the data that are collected. This interaction can lead to data that are more representative of the lived patient experience, but, he cautioned, these data may also be noisier due to a variety of confounders (e.g., geographical location).
Advantages of Measurements Collected “in the Wild”
To illustrate the opportunities of collecting data “in the wild,” Omberg compared performance on multiple measurements from mPower,2 a large Parkinson’s disease (PD) study. In this study, the variation in performance
___________________
2 mPower was a 3.5-year mobile research study developed by Sage Bionetworks that explored how the progression of PD may be unique to individuals. More information about mPower is available at https://parkinsonmpower.org (accessed May 9, 2020).
across time for individuals was greater for people with PD than for healthy controls. To illustrate the consequences of this for a typical clinic-only protocol Omberg showed the measurements of tapping-speed data3 from a single individual, with PD collected through traditional in-person assessment and through a DHT. Over a 6-month period, data on the individual’s tapping speed were collected each day using a smartphone app as well as during three in-clinic visits. The data collected through a DHT showed substantial variability in the individual’s performance over the study period, with a slight rising trend in the number of taps over time. In contrast, data collected during three clinical visits showed a comparatively drastic upward trend. These results may be partially due to the fact that only three data points were collected in-clinic, he said, but they could also be due to the timing of the data collection. Patients may be more likely to come to the clinic when they are feeling well, as opposed to performing the tapping measurement at home on a daily basis, even if they are feeling poorly.
Considering Context to Identify Confounders
It is important to consider contextual factors in research based on remote-sensor data in order to identify and account for potential confounders, Omberg said. For example, when trying to measure the impact of a disease or intervention on a phenotype using a smartphone or wearable device, the phenotype and the disease or intervention can interact and affect each other. Both can also be affected by a range of confounders, such as the location, timing, or duration of the individual’s smartphone or device use. Even the weather can be a confounding factor, Omberg explained, and provided an example of a study on multiple sclerosis, which found that weather has a strong effect on a person’s disease phenotype.
The context of the study itself also matters, Omberg said. Compared to well-controlled clinical trials, large-scale observational studies with open recruitment tend to be more vulnerable to selection bias. For instance, an observational study using real-life data from a PD mobile health study, mPower, was intended to build a classifier of PD status, but most of the volunteer participants were young and healthy (Neto et al., 2019). If not accounted for, this type of confounding variable can have a large effect and lead to overestimation of the signal, Omberg said. In mPower, for example, the age distribution of the participants caused a classifier for PD (e.g., a diagnostic algorithm) to primarily learn age-related signals rather than disease signals.
___________________
3 Tapping speed is used as a measure of bradykinesia, a primary symptom of PD characterized by slowness of movement.
Well-designed studies can also be subject to confounding effects, Omberg said. Even if there is no bias at the start of a study, there may be bias when the data are analyzed at the end of the study. One meta-analysis evaluated indicators of participant retention across 8 studies and 109,000 participants and found variation in who enrolled and how long they stayed in the study (Pratap et al., 2020). The group conducting that research found that older people tend to stay in studies longer and that the return of value to participants can affect whether they stay enrolled in a research study (i.e., people with a disease may see more value in participating in a research study than those without a disease). In fact, Pratap et al. (2020) found that participants with the disease being investigated tended to spend roughly twice as long in the study than participants without the disease. The white-coat effect was found to be another potentially confounding factor: people tended to stay enrolled around 4.5 times as long when the idea of participating was introduced to them by a physician.
The identity confounder can also contribute to an overestimation of a study’s success. Omberg explained that in protocols that use digital health, much larger volumes of data are collected from single individuals than in other types of protocols that collect measurements more infrequently. Therefore, the possibility of autocorrelation must be considered in DHT datasets that involve repeated measurements from a single individual. Furthermore, each individual measurement cannot be treated as an independent measurement, he said. A review of 47 digital health studies found that half of them had ignored this correlation structure in their data (Saeb et al., 2017) and that this led to a large underestimation of errors (Chaibub Neto et al., 2019). Substantial ethical concerns can also arise when the data are sufficiently high-dimensional to build models and predictors that represent the identity of specific individuals in a study, not just their disease characteristics.
Another contextual consideration relates to validation, Omberg said. Multiple avenues of validation can help to determine whether measurements collected in and out of the clinic are concordant; this is important because not all measurements are translatable, he said. To illustrate how in-clinic and out-of-clinic measurements can differ, he shared findings from a validation study (Webster et al., 2020) that was designed to identify a simple and inexpensive measurement of cardiorespiratory fitness to be deployed in All of Us. The validation study used two different VO2max4 protocols: a 3-minute step test (3-MST) and 12-minute run test (12-MRT). Both protocols were used to collect data in and out of the clinic, which
___________________
4 VO2max is a measurement of the maximum rate of oxygen a person consumes during intense exercise.
consisted of a single in-clinic measurement for each protocol and multiple non-supervised, at-home measurements for each protocol collected via smartphone. For the measurements collected in the clinic, the two protocols were relatively concordant (3-MST: 0.61; 12-MRT: 0.66). This level of concordance is not sufficient for basing clinical decisions on, but it is useful in collecting large volumes of survey data from many participants, Omberg said. The at-home measurement using the 3-MST protocol was translatable, with a concordance of 0.61. However, the 12-MRT protocol failed in the at-home measurement component, with a concordance of just 0.25. This demonstrates the importance of testing whether measurements are translatable outside of the clinic, he said.
Opportunities to Improve Health Insights from Real-World Sensor Data
Omberg outlined several opportunities to address challenges related to confounders and validation in order to improve health insights derived from real-world sensor data. Data from digital health research needs to be shared in a way that is ethical but also makes the data available and accessible to a broader set of stakeholders, he said. For example, Synapse5 is a repository for sharing data collected using smartphones and wearables. Synapse also houses analytical tools for processing and analyzing DHT data and other mobile health resources. The mPower researcher community has benefited from this type of ethical data sharing. The data have been accessed by hundreds of individuals and institutions, leading to several dozen publications thus far (Bot et al., 2016).
Beyond making data more widely available, frameworks should be built to encourage people to work on the data and compare their findings with each other using impartial benchmarking methods, Omberg said. One strategy is to create a challenge for researchers by posing a problem with a new dataset and fostering competition among participants from different sectors, then asking the competitors to collaborate to interpret the differences between their models after the competition. For instance, Sage has hosted a couple of challenges for building methods for predicting disease severity from accelerometer data. The mPower project recently ran a challenge on predicting disease and severity for PD. The performance of the default model was far surpassed by the top winning models, Omberg said. He added that the challenge has changed the way that the participant institutions conduct their processing of this type of data. This illustrates how such open, collaborative methodologies can promote the
___________________
5 The Synapse resource is available at https://www.synapse.org/digitalhealth (accessed May 9, 2020).
development of reusable, broadly accessible tools, code, and pipelines.6 Another example of this type of effort is the Open Wearables Initiative,7 a collaboration designed to “promote the effective use of high-quality, sensor-generated measures of health in clinical research through the open sharing of algorithms and data.” Together with several other institutions, Sage Bionetworks is building a community hub and DHT registry for algorithms and data, and also developing a benchmarking program to evaluate those algorithms.
Value of Participatory Approaches to Research
In studying specific diseases and working with patient communities, it is important, Omberg emphasized, to collaborate with patients before designing a study protocol. Participatory approaches can be used to seek input from patients and the public at large about what sensors could or should measure, which can help to ensure that studies are tracking the outcomes that are most personally relevant to individuals. Participatory approaches might involve convening a patient group to act as advisors and conducting design exercises with patients to understand the burden of disease and the issues that are of the greatest concern for patients. Engaging with research participants can inform a study design and ensure its value to that community, while also obtaining the measurements the clinicians or researchers are seeking. Through this process, researchers typically adapt what they want to measure based on input from the patients, Omberg added.
DIGITAL DATA COLLECTION BY THE ALL OF US RESEARCH PROGRAM
Chris Lunt, Chief Technology Officer, All of Us Research Program
All of Us is an innovative research effort, launched by NIH in 2015 with the aim of collecting data from at least 1 million people in the United States. All of Us additionally strives to include participants from races or ethnicities that have been historically underrepresented by medical research. According to Lunt, as of March 2020, more than 250,000 people had joined the project, 80 percent of whom are underrepresented in some respect—for example, based on their access to care or socioeconomic status
___________________
6 Examples of these tools are available at https://github.com/Sage-Bionetworks/mhealthtools (accessed May 9, 2020).
7 Information about the Open Wearables Initiative is available at https://www.owear.org (accessed May 8, 2020).
(Kaiser, 2019). The ethos of All of Us is to engage with participants as partners in the research project. There is a strong focus on returning value to the participants in the program, and the project is built to be a longitudinal study that enables participants to be re-contacted over time. Multiple data types are being collected, including electronic health records (EHRs), surveys, baseline physical measurements, biospecimens, and genomic data. Because All of Us is intended to be an open, national-level resource, Lunt and his colleagues are developing open-source software and tools to ensure that the data are accessible to all researchers, including citizen scientists. He emphasized that the project is hypothesis-neutral and broadly useful because it is not intended to serve any particular audience exclusively.
Lunt and his colleagues are working to incorporate DHTs in their data collection as part of All of Us. As a starting point for integrating DHTs, All of Us created a “bring-your-own-device” (BYOD) program that allows participants to connect a Fitbit or Apple Health app. Pilot projects for specific smartphone-based apps for collecting data on participants’ cardiorespiratory fitness and mood are currently under way, Lunt said, with other apps also in development. While All of Us started with a BYOD strategy to enable earlier progress, the next step will be to distribute Fitbits to those participants who do not already have a device, Lunt said.
Value of Data Collected Through Digital Health Technology
Lunt discussed the value of data collected through DHT. From his perspective, he said, the greatest value of this type of data is the ability to capture information about participants when they are outside of the clinic. As discussed by Omberg, the intermittent nature of in-clinic data collection and the environment of the clinic can create a host of confounding factors. Because DHTs allow for longitudinal, intensive, and repeated measurements, they enable the collection of greater volumes of data than other strategies. Furthermore, the passive data collection enabled by DHTs can alleviate the burden on participants who are taking part in a longitudinal study that collects large amounts of data on many variables. Another advantage is that DHT data are not as susceptible to the self-reporting biases seen with data collected through survey instruments, Lunt said. Use of DHTs for data collection can also reduce costs, because it builds on an existing and expanding infrastructure of research and device development; integrating DHT data does not necessarily warrant additional investment.
However, the use of DHTs for data collection has several drawbacks, Lunt said. DHT data tend to be narrow and generally deductive, in that they build on existing hypotheses (e.g., the premise that “steps matter” and thus it is useful to measure a person’s step count). The technology
often relies on external infrastructure, such as mobile phone networks, which can skew the data collected. Certain security risks associated with DHTs may require negotiating with technology providers or other external partners in a way that involves yielding some degree of control, Lunt added. For example, participants who wish to share their Fitbit data with All of Us are directed through a process of granting consent for data sharing over which Fitbit maintains strict control and is not modifiable in any way by All of Us. Similarly, participants can modify the types of data that can be read by program researchers simply by changing the settings within the Apple Health/HealthKit smartphone app. Because All of Us has no control over which types of data a participant shares, it is possible for a participant to unknowingly unshare data that the program is specifically focused on collecting.
Criteria for Prioritizing Assessment and Device Strategies
Lunt explained that in considering how to make investments in DHTs, All of Us developed a set of prioritization criteria for choosing a particular assessment or device strategy: (1) science, (2) recruitment, (3) engagement, (4) partnership, (5) cost, and (6) logistics. The first criterion is the extent to which the strategy helps advance the scientific agenda of the program. The second, recruitment, evaluates how well the strategy will integrate into an existing audience. For instance, an advantage of working with outside providers, such as Apple and Fitbit, is that participants will often have years of prior data collected by those providers that they may be willing to share with All of Us. The engagement criterion assesses whether a strategy will ensure that participants feel valued and stay interested in engaging with the program. Returning information to the participants and providing them with insights about their own health can help foster engagement and interest, he noted.
All of Us has many external partners (e.g., Blue Cross Blue Shield, Walgreens), so DHTs should provide ways for partners to contribute to the program, Lunt said. Costs, the fifth criterion, include monetary costs and the costs of program oversight and program attention (e.g., engaging with institutional review boards and regulatory authorities that can be bottlenecks for the research program). Another cost to consider is the burden on participants in terms of their time and ability to engage in and understand DHT collection. All of Us is a national program, so logistical concerns relate to accuracy and the ability to access people in indirect ways, such as through the mail or through a navigator, because the program could never have direct physical access to all participants. Each of these six criteria has been further broken down into sub-criteria by All of
Us for considering a strategy, Lunt said. Box 3-1 details the specific questions used by All of Us to evaluate the scientific value of a data type.
Value of the Bring-Your-Own-Device Strategy
Lunt described some of the advantages and drawbacks of the initial BYOD strategy adopted by All of Us. Around 77 percent of people already have smartphones and about 12 percent of people—and up to 30 percent in some segments—already have their own wearable devices, he said. Smartphones tend to be the primary communication and wearable device for many families regardless of socioeconomic status. Advantages of the BYOD strategy have included the ability to immediately engage with the audience, the lower cost, and the potential for participants to share their pre-existing data. As of March 2020, around 7,000 Fitbit users had connected to All of Us, Lunt said. Substantial amounts of data are available to the program from around 2018 onward, with some participants sharing data that stretch back to 2011. A disadvantage of the BYOD strategy is that the data collected can be skewed due to participant self-selection and participants’ use of different devices. Other drawbacks include a limited audience and the curation costs required to collect a limited set of data.
Digital Health Technology Strategy for the All of Us Research Program
Lunt outlined five elements of All of Us’s core strategy for collecting data through DHTs. Initiating and maintaining participant engagement have been major challenges encountered in attempting to conduct a longitudinal strategy for a diverse audience. The intent is to develop a long-term, cross-component pipeline of pilot studies and tests that can be used over time, while also balancing those efforts with thesis-driven selection. For example, All of Us has identified morbidities related to cardiorespiratory fitness as a priority area of focus, so it will be valuable to connect with Apple HealthKit, an app that many people are already using. Another core component is a commitment to using off-the-shelf consumer technology. In part, this is motivated by the size of the research program and the consequent large investment that would be required to develop a new dedicated device for the program.
ADOPTION OF DIGITAL HEALTH TECHNOLOGIES
Alicia Staley, Senior Director of Patient Engagement, Medidata Solutions
Staley’s remarks were framed by her perspective as a three-time cancer survivor with more than 30 years of survivorship. Three decades of paper-based and DHT data have been collected from her across different health systems in multiple states, she said, but there is not yet a single, comprehensive way for her to look at her medical record in its entirety. This contrasts with the relative ease with which she has been able to capture her own health data using DHTs over the previous 2 years, she said. Through the use of the Oura Ring,8 she has collected comprehensive data about her heart rate, respiration, sleep, and activity level, which have strengthened her ability to manage her own health. For instance, based on increases in respiration rate and body temperature, she is able to predict when she is about to develop a cold with a high degree of certainty; this enables her to take preventive action, such as rest and hydration. She drew attention to the ongoing TemPredict Study,9 which is gathering data collected from Oura Rings used among frontline workers in the coronavirus disease 2019 (COVID-19) pandemic, and could serve as a catalyst to increase the use of DHTs in clinical care settings and in clinical research.
___________________
8 Information about the Oura Ring is available from https://ouraring.com (accessed May 9, 2020).
9 More information about the TemPredict study at the University of California, San Francisco, is available at https://ouraring.com/ucsf-tempredict-study (accessed May 26, 2020).
A Patient’s Perspective on Clinical Care Versus Clinical Research
Staley offered a patient’s perspective on the distinction between clinical care and clinical research and reflected that patients who engage with either of those settings tend to view those experiences as a single snapshot of their own care journeys. “They do not see it as clinical research … or clinical care; they view it as health care,” she remarked. When DHTs are used in clinical research and in clinical care, the technology’s use can seem like fragmented touchpoints to patients. Lack of coordination between these technologies may contribute to the issues with adoption of DHTs by patients, Staley suggested. Furthermore, the clinical perspective on these technologies tends to be siloed—that is, the technology is considered to be either a care option or a research option. In contrast, patients tend to view the technologies as a single point of interaction with their overall health care team. Historically, major pain points from the patients’ perspective have related to accessing, sharing, and transferring their own health data when it is important not only for the patient but also for the clinical care interaction or the clinical research interaction. The expansion of EHRs and the ability to transfer information more easily have helped to reduce those barriers for patients, Staley noted.
Barriers to Patient Use of Digital Health Technologies
To increase the uptake of DHTs in clinical care and research and increase the volume of data collected, Staley suggested finding ways to take advantage of the consumer behaviors driving the adoption of commercial mobile health technologies that are already pervasive. For instance, she said an estimated 120 million Apple Watches10 have been used at some point in the previous 1.5 years. Similarly, as of January 2020, Fitbit reported that it had sold a total of 100 million devices, with 30 million currently active users.11 Staley remarked that while it is clear that consumers are buying and using DHTs, these technologies have not yet extended comprehensively into a clinical research setting. All of Us and smaller-scale pilots could potentially fill this gap, but larger-scale adoption on the clinical research side is still needed to tap into the mainstream consumer mindset and behaviors, Staley said. The COVID-19 pandemic has exposed the need for more creative ways to capture data and maintain clinical trials in virtual settings. She suggested that consumers and
___________________
10 More information about the Apple Watch device is available from https://www.apple.com/watch (accessed May 9, 2020).
11 More on Fitbit’s sales and active user figures is available in a press release. See https://investor.fitbit.com/press/press-releases/press-release-details/2019/Fitbit-to-Be-Acquired-by-Google/default.aspx (accessed June 22, 2020).
patients should be engaged to identify additional points at which data could be captured for building on pervasive technologies as a way to help support clinical research. Innovative strategies for policy adoption and consumer adoption will be needed if these tools are to be used to their full potential, however.
In addition to the lack of full-scale adoption in clinical research, Staley said that another current barrier is the lack of data sharing from consumer mobile health technologies to health care providers. For instance, she cannot easily and directly share critical information from her Oura Ring about her respiration with her primary care physician or oncologist. Instead, she must track her own data, analyze them herself, and bring the data profiles to her providers.
The technology divide poses another barrier, Staley said. In her experience as a member of the breast cancer patient community, she said, she has long been aware of the digital divide and its potential to create a bias toward healthier populations with higher education levels when data are collected from consumers in BYOD studies. She cautioned that with the advent of so many new tools, it is important to be mindful of populations that may be left behind. Many households and even entire communities lack consistent, reliable Internet access. “We need to be able to educate our users on these tools, make sure that they have access to these tools, and utilize them in a way that makes sense for their life,” she said. One strategy is to find patients who are well respected in their communities and are willing to step forward, use tools, and take advantage of clinical research opportunities. These patients can then become beacons of hope and information for other patients in efforts to integrate DHTs into clinical trials and to promote clinical trial awareness in general. Empowering those patients to be voices for their own communities can also help bridge the technical and community divides that have been evident in clinical trial settings, Staley said.
Integrating Digital Health Technologies into Clinical Trials
Staley explained how Medidata is working to adapt the clinical trial platform to include patient data. An entire team at the organization is focused on integrating mobile sensors and other types of digital technologies into clinical trials. There is value in working with organizations that have existing, validated tools that can be integrated into a clinical trial, she said, adding that more companies are beginning to conduct pilots or smaller-scale studies that are integrating technologies approved by the U.S. Food and Drug Administration. Another positive trend is increasing buy-in among consumers and patients as companies such as Apple are starting to launch their own studies that use DHTs. Medidata has also
conducted a number of studies on the use of DHTs in clinical trials, with the results made publicly available.12 As more companies conduct trials designed around patient-focused outcomes and goals, DHTs will increasingly be integrated into larger-scale clinical trials, Staley predicted.
DISCOVERY THROUGH PERSON-GENERATED HEALTH DATA
Luca Foschini, Chief Data Scientist and Co-Founder, Evidation Health
To explore opportunities for PGHD to inform participatory approaches to the discovery and development of health interventions, Foschini offered examples of research conducted by Evidation Health. He explained that PGHD enables continuous monitoring of health outcomes at the individual level so as to make it possible to better understand and measure a person’s experience. PGHD typically comprises a dataset that is collected either by a person or their caregiver to qualify and address the person’s health. The acronym is often defined as “patient-generated health data,” but because much data can be collected from the individuals before they become patients, Foschini said he prefers “person” to “patient,” he said. This type of long-term data collection can be drawn from a variety of sources, including smart clothing, wristbands, and smart houses and automobiles (see Figure 3-1). He added that a large component of the PGHD that can be collected today comes from the voice of the person, through the direct articulation of subjective feelings and experiences.
Leveraging Person-Generated Health Data to Inform Public Health Interventions
PGHD makes it possible to carry out universal research on how individuals feel, function, and survive, Foschini said, because the data can be collected remotely and on a large scale. He suggested that collecting PGHD should be the first step in building health interventions—from the development of drugs and devices to public health policies—and in understanding how those developments affect the people they are ultimately intended to serve. He used the COVID-19 Pulse study13 as an example of how this type of participatory discovery and development can inform public health interventions in real time. As of March 2020, Evidation had recruited more than 100,000 participants from among its Achievement program
___________________
12 More information about the studies is available from https://www.medidata.com/en (accessed May 17, 2020).
13 Information about the COVID-19 Pulse study is available from https://evidation.com/news/covid-19-pulse-first-data-evidation (accessed May 17, 2020).
NOTE: ECG = electrocardiogram; NO = nitric oxide; O2 = oxygen; UV = ultraviolet.
SOURCES: As presented by Luca Foschini, March 24, 2020. Originally from Gambhir et al., 2018.
participants for an ongoing longitudinal study covering almost 90 percent of counties in the United States (see Box 3-2 for more information about the program). The aim of the study is to understand how people in the United States are coping with the COVID-19 pandemic by collecting data about how their behaviors are disrupted and how their perceptions are changing. These data can offer insights into whether interventions are working as well as into how they are perceived to be working. As of March 18, 2020, about two-thirds of participants reported washing their hands more frequently over the previous week, but only about one-third reported avoiding large gatherings. Around 40 percent reported increased anxiety over the previous week. Financial anxiety is generally mediated by social determinants of health, Foschini said, and in the United States, people without health insurance tend to have much greater levels of anxiety. Uninsured individuals are also more likely to present at an emergency room if they have symptoms of COVID-19 than are patients with insurance, who tend to visit their primary care provider first. Understanding how populations will react based on their health insurance status is important for building interventions to curtail contagiousness at the point of care, he said. This PGHD can also be used to visualize the impact of the mitigation strategies being deployed across the country. For instance, Evidation was able to access wearable device data of respondents who consented and plot those data over time. The analysis of those data revealed variability in the impact of public health strategies in different states. In some states, participants’ data indicated decreased mobility (measured as Fitbit steps) compared to baseline in the days after the United States declared COVID-19 a federal emergency.
Using Person-Generated Health Data to Complement Real-World Data
PGHD from a commercial wearable device can also serve as a complement to traditional real-world data, Foschini said. For instance, continuous data from a commercial wearable can be used to monitor post-operative recovery from surgery. Foschini described a study that surveyed almost 51,000 people enrolled in the Achievement program about whether they had undergone a medical procedure or surgery in the previous year (Ramirez et al., 2020). Of the 1,203 respondents who reported having undergone a weight loss procedure, 675 had some Fitbit data and 118 had high-quality, high-density Fitbit data that they consented to share. There is a large decrease in available data when data-quality constraints are applied, he noted, suggesting that researchers should be mindful and standardized in defining what “data quality” means in the context of PGHD. The researchers found large changes in patients’ activity levels at 12 weeks compared with baseline (pre-surgery) in the measures captured by the patients’ Fitbit devices. Overall, the patients’ mobility increased to above baseline levels after the expected dip in mobility immediately after the surgery. Interestingly, the patients’ resting heart rate dropped drastically by six to eight beats per minute over the 12 weeks. Total sleep time increased as expected and then decreased back down to baseline, but sleep efficiency as measured by Fitbit increased and was maintained over a longer time. This illustrates how real-world measures can contribute to a broader understanding of how patients feel about their recovery from surgery, Foschini said. Currently, the main measures of success for a weight loss procedure, beyond weight loss itself, are outcomes related to insulin resistance reversal or complications. However, the patient may care more about outcomes such as sleeping better, walking more, or better fitness conditioning that can be captured using real-world PGHD.
Ways Forward for Person-Generated Health Data
Foschini outlined opportunities to expand the use of PGHD to inform participatory discovery and rapid development of interventions that have the individual at the center, ranging from public health policies to drugs, devices, and digital therapeutics. The current ability to build trusted relationships with individuals to collect PGHD is unprecedented, he said. For instance, enrollment recently started in the Heartline study,14 which will look at how DHTs such as the Apple Watch and a health program delivered through an iPhone app can improve cardiovascular disease
___________________
14 More information on Johnson & Johnson’s Heartline study is available at https://www.heartline.com (accessed May 17, 2020).
outcomes in 140,000 older people in the United States. Experiences and lessons learned in trying to deploy and run large-scale efforts involving DHTs have identified several strategies for moving forward, Foschini said. The first is to ensure interoperability. Although The Office of the National Coordinator for Health Information Technology and the Centers for Medicare & Medicaid Services recently released its Interoperability and Patient Access final rule,15 no equivalent standard has been set for PGHD specifically, Foschini said. All stakeholders—including individuals, providers, and regulators—should have a common data format for storing and transmitting PGHD in order to share data collected using DHTs (e.g., sharing Oura Ring data with a provider). He noted that the Smart Markers16 domain is working to include PGHD into the Smart Framework and urged the community to support this effort while the setting is still precompetitive. PGHD is as identifiable as DNA, he said, so good governance will be needed to maintain people’s privacy and security. Ethical concerns will abound because PGHD straddles research, care, and consumer experiences, he added. Ensuring representativeness and striving to reduce the technological divide will help to ensure that all people can share in the benefits of PGHD. Developing a common analytical framework will also be important, he said. To improve the quality and density of data, there should be a standardized analytic pipeline that allows researchers to evaluate whether data collected in the real world are fit-for-purpose for a given type of analysis.
DISCUSSION
Developing Operational and Analytical Standards for Digital Health Technologies
Given the diversity of tools and activities under way in this space, Vayena asked the panelists if it would be beneficial to work toward developing a novel set of standards for digital technologies, either for the way that they operate or how they are used. The raw accelerometry data collected by a Fitbit device is not shared with the user, Lunt noted; instead the user receives information that has been curated by Fitbit. He suggested that instead of focusing narrowly on a single measure—which is necessarily reductive of the underlying raw data—it could be useful to
___________________
15 More information about the Centers for Medicare & Medicaid Services Interoperability and Patient Access final rule can be found at https://www.cms.gov/Regulations-andGuidance/Guidance/Interoperability/index (accessed May 31, 2020).
16 Information about the Smart Markers domain is available from https://smarthealthit.org/smart-markers-a-framework-for-patient-generated-data (accessed May 17, 2020).
create an environment that encourages multiple stakeholders to interpret those raw accelerometry data in different ways.
The large body of data collected from digital tools over the past several decades is largely inaccessible to the public, partly due to the financial value of these data, Omberg said. This makes it difficult to evaluate the accuracy of the algorithms and outputs of DHTs commonly in use and underscores the need to establish methods for independent benchmarking. For instance, he said, in his experience using the Garmin and Fitbit accelerometers, he has found that their step counts can be discrepant by up to 30 percent. He noted that the actual algorithms also have value, not just the data. His organization is working with the Open Wearables Initiative to develop independent benchmarks of algorithms by running them on subsets of data. This type of modeling does not require an entire dataset to be shared publicly.
There is an opportunity for adding a layer of standardization for data storage and data sharing, Foschini said. Currently, different platforms for data collection (e.g., Apple HealthKit, Fast Healthcare Interoperability Resources within EHR systems, or Clinical Data Interchange Standards Consortium/Clinical Data Acquisition Standards Harmonization) do not have a common way to represent data collected in a multivariable time-series format. Developing a common way to store and transport those data would make it easier for platforms to run their own algorithms and conduct their own analytical validation on the data that are being collected by a third party, Foschini said. He suggested that an abstraction layer could be built into analytical validation to foster collaboration in this domain as has happened in other research communities, such as the genetics community. That community took 20 years to reach a consensus on the pipeline from raw sequencing data to single-nucleotide polymorphism studies, but it is now an established standard.
Given the large volume of data collected through digital modalities, Lunt suggested shifting the paradigm from “moving the data around and keeping the tools in place to moving the tools around and keeping the data in place.” This transition is already under way, although developing the necessary structures and ensuring data privacy will require substantial effort. A workshop participant asked if there is an ultimate authority that can advise the community about whether a given DHT is validated. Lunt said his organization works closely with the UK Biobank,17 which has shared useful lessons gleaned from its long history of doing this type of work. One takeaway is to engage earlier with different audiences to gain various perspectives on which measures are considered to be the
___________________
17 Details about the UK Biobank resource can be found at https://www.ukbiobank.ac.uk (accessed May 17, 2020).
gold standard. For instance, the UK Biobank ran a study for 10 years before granting any access to its data, at which point the cardiology community suggested that a more appropriate measure should have been used for some of the data. A challenge will be finding a way to query across an entire community of researchers to understand what measures the various researchers consider to have the greatest predictability, which will likely be in constant flux as sensors and devices improve.
