5

Digital Health Technologies for Pivotal Trials

The workshop’s third session focused on the use of digital health technologies (DHTs) in pivotal trials,¹ a crucial phase in the drug development process that generates the evidence on which regulatory approval decisions are based. Sean Khozin, global head of data strategy at Janssen Research & Development, LLC, provided an industry perspective on the potential impact of DHTs on the velocity, volume, variety, and veracity of data. He also described technical and procedural challenges encountered when incorporating data collected from DHTs into pivotal trials. Ritu Kapur, head of biomarkers at Verily Life Sciences, provided a practical overview of the processes of signal verification, analytical validation, and clinical validation for novel DHTs. She also highlighted several opportunities to improve the measurements captured by those technologies going forward. Leonard Sacks, associate director of clinical methodology in the Office of Medical Policy at the Center for Drug Evaluation and Research at the U.S. Food and Drug Administration (FDA), offered a regulatory perspective on ways that DHTs could enable decentralized clinical trials, as well as on their potential to capture novel measurements. The session was moderated by Husseini Manji, global therapeutic head of neuroscience at Janssen Research & Development, LLC.

INDUSTRY PERSPECTIVE ON DIGITAL HEALTH TECHNOLOGIES IN PIVOTAL TRIALS

Sean Khozin, Global Head of Data Strategy, Janssen Research & Development, LLC

Khozin said that the expanding universe of big data presents complexities regarding four computational dimensions (see Figure 5-1):

• Volume: Dataset size (e.g., computing storage sizes—megabyte, gigabyte, terabyte, and petabyte)
• Variety: Data type (e.g., tables, databases, clinical trial data, electronic health records)
• Veracity: Data noise and uncertainty (i.e., where data lie on the continuum of being structured to undefined)
• Velocity: Data flow and processing speed (e.g., batch, intermittent, near real time, and real time)

The expanding complexity that big data poses is depicted in Figure 5-1. Whereas data in the center of the concentric circles represents a hypotheti-

___________________

¹ A pivotal trial is held to the highest standards of rigor and quality control; it requires prespecification of all components (e.g., study design, enrollment, dosages, comparators, measures, endpoints, statistical plan) in discussion with regulators prior to conducting the trial.

cal reductionist center where biomedical research does not take advantage of big data, DHTs fall on the outer edge of the concentric circles. As such, Khozin noted that the “holistic” edge of big data represents emerging opportunities to leverage multiple data types that, despite having complex characteristics with respect to data standards, quality, size, and veracity, are fundamental building blocks of developing a new generation of precision therapies with near-real-time velocity. In some cases, he added, data from DHTs (e.g., wearables) close to the holistic edge of big data can also be used to capture patients’ experiences in pivotal trials, using novel trial designs that accommodate more decentralized data collection. Due to the unique nature of data assets emerging from DHTs, he explained, it is important to consider the technical and procedural issues associated with incorporating those modalities into clinical trials.

Technical Considerations for Using Digital Tools in Clinical Trials

Khozin explained that measurements, verification, and validation are used to assess the technical features of digital tools (Coravos et al., 2019b). In order to trust the data that are captured by DHT measurements, the software and hardware specifications must be standardized and clearly understood. Such measurements typically involve three layers: an input layer (e.g., a camera, microphone, or sensor); a processing layer (i.e., an algorithm that pro-

cesses the input); and an output layer (i.e., a digital biomarker). The output layer might be a familiar clinically validated measurement (e.g., heart rate). However, an advantage of using DHTs is the ability to quantify outputs that are currently unfamiliar or unquantifiable. For example, performance status is a subjective assessment used to understand the patient’s daily activities and is used in oncology to determine participant trial eligibility and the intensity of treatment regimens (Kelly and Shahrokni, 2016). Work is ongoing to use digital tools to better quantify the assessment of performance status using digital sensors that can track a patient’s daily activities.

Verification and analytical validation are additional technical components to consider when evaluating and using DHTs in pivotal trials, Khozin said. This process includes engineering benchmarks to ensure that a product is measuring and storing values accurately. He noted that a tool’s accuracy, precision, and reliability are three related yet nearly mutually exclusive concepts. In some cases, it is possible to extrapolate experiences from how companion diagnostics are developed and analytically validated in terms of accuracy, precision, and reliability.² For example, a heart rate sensor should be able to faithfully convert electrical signals into an accurate, clinically relevant measure—in this case, heart rate in beats per minute. Such a device would then need to be analytically validated to ensure that it is accurately measuring what it is supposed to measure.

Devices that have been verified and validated analytically still need to be clinically validated, be it prospectively in a clinical trial or separately as part of a qualification program. Clinical validation addresses whether the measurement is applicable in the target population and whether the context of use renders the digital biomarker fit-for-purpose (see Christopher Leptak’s presentation in Chapter 4). Further expanding on his example of a heart rate sensor, Khozin explained that clinical validation would entail ensuring that the output of the sensor is a meaningful endpoint and could replace, for example, the traditional tactile measurement of a patient’s pulse for a clinical trial.

Digital Health Technologies as Diagnostic Tools

Khozin spoke about how the issue of false negatives and positives could be addressed when using clinically validated DHTs for diagnostic purposes. Established methods for addressing false negatives and positives in traditional diagnostic tests will also apply to DHTs. All tests used in the clinical setting, Khozin noted, have false-negative and false-positive rates, which are generally managed by purposely administering tests to

___________________

² Accuracy refers to how close a captured measure is to the true value of an endpoint. Precision refers to how consistent repeated measures are to each other. Reliability is a similar concept to precision and refers to the degree to which a measurement instrument is consistent and free from error (Trajkovic, 2008).

patients who, based on the patient’s data and the provider’s clinical judgment, have a high probability of having a target disorder. This practice, known as increasing pretest probability, can also be employed when using DHTs to make them more predictive.

Procedural Considerations for Using Digital Tools in Clinical Trials

Khozin discussed procedural considerations related to the use of these DHTs in pivotal clinical trials, including clinical validation and the design and conduct of clinical trials (Coravos et al., 2019b). To illustrate the process, he drew an analogy with how biomarkers are validated in oncology clinical trials. Biomarkers are typically validated clinically prospectively during a clinical trial and are paired with a targeted therapeutic using an analytically validated assay, rather than being separately evaluated as part of FDA’s Biomarker Qualification Program, Khozin said. The same methods can be applied to clinically validate digital biomarkers prospectively in a clinical trial, he added.

Further procedural considerations relate to the novel clinical trial design opportunities that the use of digital tools can allow, Khozin said. Decentralized clinical trials have garnered interest in recent years as a strategy for scaling studies (Khozin and Coravos, 2019). Because they are decentralized, these types of clinical trials have unique features in terms of where the data are being captured and who is collecting the data. He noted that there is a continuum of decentralization in the sense that most traditional clinical trials already have decentralized components, such as collection of data on the phone or via home visits (instead of a research facility) and outsourcing of testing to commercial laboratories rather than having it done at a centralized laboratory. Khozin said that in appropriate cases, DHTs today could allow data to be collected completely remotely, perhaps even in the absence of any intermediaries. This would create opportunities to collect data from patients where they live. Hybrid approaches could also be deployed, he added. An example of a siteless, completely decentralized clinical trial is the Heartline study, which is exploring how commercial technologies (e.g., Apple iPhone and Watch) can facilitate the early detection of atrial fibrillation.

PERFORMANCE REQUIREMENTS FOR DIGITAL HEALTH TECHNOLOGIES IN PIVOTAL TRIALS

Ritu Kapur, Head of Biomarkers, Verily Life Sciences

Verily is currently developing means to use digital measurements and emerging DHTs to improve the success of drug and medical device development. Specifically, Kapur explained, the goal is to create endpoints that

increase the efficiency of clinical trials and that are useful in the context of pivotal studies. To this end, digital measurements need to undergo a process of verification and validation to reliably demonstrate the safety and efficacy of investigational products. Specifically, this involves a three-step process of signal verification, analytical validation, and clinical validation. Kapur defined each term and gave examples of each:

1. Signal verification tests whether a sensor is working, which typically requires bench testing. Challenges can depend on the diversity of devices being used in a study. For example, a “bring-your-own-device” strategy can create a prohibitively large number of permutations of devices and operating systems for analysis.
2. Analytical validation tests whether the DHT is measuring what it is intended to measure. This usually involves measuring the underlying algorithm’s performance against a trusted corroborative device. In some cases, this can be performed through comparison with a score from a human rater. Challenges can include collecting enough naturalistic data for comparison because these tend to be noisy and variable. Furthermore, in cases where human observations are the benchmark, the accuracy of corroboration is pinned to a subjective measure.
3. Clinical validation tests a digital measurement’s predictive power in the context of its intended clinical use. This generally relies on testing a tool with datasets in which the clinical outcome of interest varies in order to see how well the test predicts a given clinical outcome, Kapur said. A challenge frequently encountered is the lack of common methods for evaluating digital metrics against traditional clinical ratings.

To further explore this three-step process, Kapur illustrated hypothetical examples of digital tools used to measure the symptoms and severity of Parkinson’s disease (PD) (see Box 5-1).

Improving Measurement by Digital Health Technologies

Kapur highlighted several opportunities to improve measurements captured by DHTs. Establishing a common approach for evaluating digital measures against subjective clinical ratings would be helpful, although how to do so remains an open question. Transitioning from subjective to objective measurements may require an agreed-upon set of performance criteria for quantitative measurements that are not pinned to subjective ratings. In the verification and validation of digital metrics, a foundational step will be to establish what counts as “good enough” performance. A common approach that is established should be agreed upon by a wide set of stakeholders, she said.

Scalability is a related issue, Kapur said. Each new digital measurement could potentially be taken through a regulatory process for approval through either a drug development pathway or a tool development pathway. However, it is not yet clear how this process could be scaled if it consisted of increasing numbers of metrics simultaneously built and combined. Balancing accuracy of validation with speed when scaling will also be important, Kapur said. In the context of building entirely new measurements, Kapur wondered whether testing an algorithm’s performance on a clinical dataset could substitute for prior analytical validation. She noted that this would require sound methodological approaches—such as appropriately separating training and testing datasets—and the ability to demonstrate that the analyses yield consistent results across multiple independent datasets.

Leveraging Digital Health Technologies for Recruitment, Retention, and Engagement

Kapur shared some of Verily’s experiences in using digital technology for recruitment, retention, and engagement. Creating a sense of human connectedness when operating at scale is a challenge encountered when DHTs are incorporated into clinical trials. Although it may seem counter-intuitive in the context of digital technology, human connection should be at the core of these efforts. For example, finding ways for participants to feel connected, such as providing help lines for people to call, can be useful when structuring rollouts. Another approach is to work with communities and learn about what helps people feel connected—particularly within successful initiatives—and incorporate those lessons learned into the technology. Kapur explained how this could be valuable by describing a collaboration that Verily had with the Radboud University Medical Center and ParkinsonNet in the Netherlands. The joint initiative enrolled participants using wearable devices to monitor their symptoms (Bloem et al., 2019). This initiative held a participant event during the study to allow participants to hear about progress of the study and allow participants to learn about each other’s experiences. This event appeared to increase engagement, and after 1 year in the study timeline, an average of 20 hours of wear time per day and a dropout rate of less than 1 percent were achieved—both of which are highly successful results for a wearable initiative.³

___________________

³ For more information on the Personalized Parkinson’s Project, see https://blog.verily.com/2019/04/visiting-personalized-parkinsons-project.html (accessed June 19, 2020).

REGULATORY PERSPECTIVE ON THE USE OF DIGITAL HEALTH TECHNOLOGIES IN PIVOTAL TRIALS

Leonard Sacks, Associate Director for Clinical Methodology, Office of Medical Policy, Center for Drug Evaluation and Research, U.S. Food and Drug Administration

Sacks noted that DHTs provide two opportunities from a regulatory perspective:

• Supporting decentralized trials: DHTs can support the remote collection of data from patients in decentralized clinical trial settings. In addition to helping ensure the continuity of clinical trials during a pandemic (see Chapter 1 for more information on the impact of coronavirus disease 2019 [COVID-19] on clinical trials) or another disaster, decentralized clinical trials provide access to patients who are unable to travel to clinical sites, offer improved patient convenience, and can address access issues among people with impaired mobility, such as people with Duchenne muscular dystrophy. Furthermore, decentralized trials provide access to people with rare diseases (e.g., inborn errors of metabolism), who may be widely distributed across the country or world.
• Supporting capture of novel measurements: Traditional clinical trials typically rely on sporadic or intermittent measurements, but DHTs can facilitate continuous measurements to capture information from participants during interim periods that would otherwise be lost (e.g., hypoglycemic episodes, falls, or seizures). Novel technologies also offer the opportunity to objectively and quantitatively measure clinician- and patient-reported outcomes, such as functional status. Functional status can now be measured by DHTs. The use of interactive task-based tests on mobile devices holds promise for enabling more frequent testing of vision, hearing, cognition, and fine motor coordination, Sacks said. DHTs can also be used to capture physiological measurements (e.g., continuous electrocardiograms, pulse oximetry, and lung function) and enable other types of novel measurements using photography.

Uses of Digital Health Technologies in Clinical Trials

Sacks described how DHTs can fit into the design of clinical trials. For example, they could be used to improve participant screening and enrichment strategies by selecting patients based on levels of disease severity or activity levels. Furthermore, DHTs could be used to refine

how performance is evaluated. During a trial, DHTs can also be used to monitor treatment adherence and drug safety as well as to provide pharmacodynamic impressions of how study drugs are working relative to their dosing. Another exciting opportunity DHTs provide in clinical trials is the potential to shape endpoints.

To explain how DHTs can contribute value in this respect, Sacks provided an overview of endpoints used in FDA pivotal trials for 280 new drug applications between 2007 and 2015. About 30 percent were approved based on clinician- or patient-reported outcomes, clinical events, or clinical signs. From a regulatory perspective, the dearth of objective ways to measure functional status represents a substantial opportunity for DHTs. Additional opportunities that DHTs provide beyond the dimension of functional status include such physiological measurements as continuous blood pressure monitoring, electrocardiograms, electroencephalograms, and pulse oximetry. Digital tools are also beginning to be used for biochemical testing, such as continuous glucose monitoring (Hirsch et al., 2019).

Measuring Functionality

Sacks explained that functional status (in terms of movement and activity) is a valuable yet challenging measure to assess across many product development areas, such as cardiorespiratory conditions (e.g., heart failure and pulmonary hypertension) and neuromuscular diseases (e.g., Duchenne’s muscular dystrophy). Functionality has traditionally been measured using a 6-minute walk test, which, Sacks noted, is a relatively crude metric that has many potential confounders. DHTs are already being used to capture more precise and quantifiable measurements of functional status. Sensors could also be used to measure functional status in the home environment through smartphone-based interactive tests of vision, hearing, cognition, and coordination, Sacks observed.

Sacks described the results of a study that measured functional status among people with heart failure (Snipelisky et al., 2017) to illustrate the capabilities that DHTs could provide. The study compared participants’ average daily accelerometry units (ADAUs) to traditional parameters (e.g., 6-minute walk test and the Kansas City Cardiomyopathy Questionnaire or KCCQ).⁴ Results of the study found a statistically significant correlation between ADAU and the 6-minute walk test, as well as between ADAU and KCCQ, across all three tertiles of study participants.

___________________

⁴ The Kansas City Cardiomyopathy Questionnaire is available at https://www.fda.gov/media/108301/download (accessed May 17, 2020).

Measuring Movement Disorders

The use of DHTs for imaging purposes also holds promise from a regulatory perspective, Sacks said. To illustrate how video technology can be applied to measure symptoms of movement disorders, he compared key results from two double-blinded, placebo-controlled studies for the approval of valbenazine, a drug for tardive dyskinesia,⁵ both of which used a 12-item clinician-rated Abnormal Involuntary Movement Scale to rate participants. While one of the studies was conducted in a traditional clinical trial setting and used a single dose of valbenazine, the other study tested two doses of valbenazine and incorporated DHTs by sending video recordings of participants to independent adjudicators who were blinded to the sequence of the video recordings and to the study drug allocations. In the first study, investigators blinded to treatment allocation who made successive subjective assessments on their patients reported improvement from baseline in placebo-treated patients. However, when investigators using video recordings were blinded to treatment allocation and to the sequence of visits, no change from baseline was observed in placebo recipients. Sacks speculated that the blinding of the sequence of visits using DHT may have removed a subjective bias in successive evaluations.

Regulatory Considerations

Sacks outlined some of the regulatory considerations that pertain to using DHTs in clinical trial settings. Verification and validation data are important for understanding whether a DHT has met its technical specifications and provides precise and accurate results in the study population. The verification and validation process could involve comparison with measurements made visually or other reference methods. Sacks said that it would be important to identify potential confounders of measurements made by DHTs. User testing would be critical to preempt operational problems during the trial and to ensure that participants are comfortable using a particular DHT. Another consideration is the justification of novel endpoints made possible by DHTs, he added. This will likely involve comparisons with existing benchmarks of drug efficacy and consultations with patients, caregivers, disease experts, and regulators. In general, he said, determinations of the suitability of a given DHT in a clinical trial are made independently of whether that technology has been cleared by FDA’s Center for Devices and Radiological Health. Informed consent dis-

___________________

⁵ Tardive dyskinesia is a condition that affects the nervous system and causes repetitive involuntary movements; it is often caused by long-term use of neuroleptic drugs to treat psychiatric conditions.

cussions with participants will also be important to determine their expectations of privacy, potential physical risks, and considerations related to real-time safety monitoring. Issues related to data custody, access controls, audit trails, and the preservation of source information may also need to be addressed, Sacks added.

DISCUSSION

Analytical Validation Without a Gold Standard Reference

Kapur and Sacks discussed approaches that could be used to perform analytical validation on DHTs in the absence of a gold standard reference measurement. Kapur suggested a combined approach to addressing this challenge—particularly for novel measurements—before the widespread introduction of a tool. Internal work within a company could involve validation in the clinical setting as well as testing using different datasets to demonstrate whether a novel measurement has less variability than current metrics. Kapur noted that data from some populations are variable in expected ways that must be taken into account. For example, people with more advanced PD are less mobile than those with early-stage disease. External work may involve convening stakeholders, such as FDA, the National Institutes of Health, and patient advocacy groups, to agree upon standards for establishing the reliability of novel measurements. Sacks remarked that dealing with a novel measurement in the absence of a gold standard, or any standard at all, is a multidimensional endeavor that requires creativity and offers opportunities for progress and collective thinking. A new measurement must be evaluated by a broad variety of constituents—including patients, caregivers, doctors, and regulators—so that a community-based decision about whether it is valid can be made. Certain features captured by a new measurement may add richness to the data. In situations where an already approved drug is known to be effective, it may be helpful to evaluate whether a new measurement could provide greater discrimination of the treatment effect than existing metrics.

Collaborative Approaches to Improving Interoperability

The panelists considered collaborative approaches to improve interoperability among DHTs and address the challenge of device heterogeneity. Kapur emphasized that addressing this issue will require an ecosystem-based solution rather than a technical one. She highlighted the benefit of having stakeholders from across sectors and disciplines coming together to collaboratively define a set of clearly outlined standards or values for DHT development. If there were a set of clear standards or values in

place, then there would be an impetus for DHT developers to make their products compliant, she added. As of now, smaller start-up companies trying to develop DHTs are often left guessing about what the standards and values should be because these have not been clearly established. The clearer the standards and values for what DHT developers should be aiming to achieve, the easier it will be for all stakeholders to participate at scale, she added.

While standards would be welcomed by the DHT developer community, Kapur made the point that standards may vary depending on the technology itself and the context of use. Sacks suggested that key stakeholders could identify specific areas of opportunity. For example, certain technologies, such as mobile phones, have clearly defined standards and are highly interoperable. Sacks added that common standards are also useful because they make it possible for the digital health community to evaluate which technologies might be suitable for a proposed use and which are not. For example, technological standards for mobile phones and smart watches may allow study participants to use their own devices rather than using a study-assigned mobile phone or wearable devices.

This page intentionally left blank.