5

Using Machine Learning to Forecast and Improve Clinical Outcomes and Healthy Aging Using Sensor Data

Alvin Rajkomar¹

INTRODUCTION

Our understanding of health and aging comes from snapshots of measurements collected in healthcare settings, such as yearly blood testing for glucose, or responses to antidepressants measured episodically every few months by a clinician. Yet the vast majority of people’s daily experiences unfold outside the eyes of the healthcare system, leaving habits, dietary choices, sleep, environmental, and social exposures unmeasured, along with important outcomes that are hard to collect with a questionnaire in a physician’s office, such as daily perception of how they feel, functional independence, and emotional state.

By analyzing real time locations and speeds of cars, apps can automatically detect traffic and re-route you to your destination to arrive sooner. It seems natural that if a system could collect lifestyle habits of millions of people through ubiquitous sensors, such as those in cell phones, and follow what happened to them—whether they developed diseases or disability—then it could direct people how to live better to reduce the risk of diabetes or to inform how we can promote an aging parent to live safely at home, effectively re-routing their life to a longer, independent life. At a high level, we are all on the same journey of aging, and while young we generally rebound back to our expected levels of functioning after illness, accidents, or life-events, but as we age, we lose our ability to return to our

___________________

¹ Google LLC, Mountain View, California. Address correspondence to: alvinrajkomar@google.com.

prior function after increasingly small stressors and physiological insults (Clegg et al., 2013). Finding the path that maintains health and robustness of individuals and populations is therefore a universal need.

However, the optimism that large data sets and complex data analysis can help us learn personalized insights to optimize our way of living to promote personal betterment or graceful aging must be tempered with the humility that this endeavor is exceedingly difficult.

The amount of data collected from individual participants in trials already exceeds the ability of a human expert clinician to review, evaluate, and interpret, and machine intelligence plays a pivotal role for analysis. The question is how can researchers thoughtfully apply best practices in machine learning (ML) and clinical research as they use data to forecast progression of aging and clinical trajectories and identify ways to improve patient outcomes.

This chapter will begin by reviewing the core aspects that constitute an ML system: input data, desired outputs, and generation of training and test data. Following this review, the chapter will discuss ways in which ML can be applied to sensor data gathered in clinical trial settings as a means of identifying potential outcomes, forecasting health trajectories, and developing interventions to improve health for older adults.

MACHINE LEARNING CONSIDERATIONS

Overview of Machine Learning

The details of ML were recently summarized (Rajkomar, Dean, and Kohane, 2019). This chapter will focus on the most commonly used type of ML, referred to as supervised ML. While supervised ML is featured here, other types of ML have been used for proof-of-concepts (Fisher et al., 2019) and show promising results.

Supervised ML differs from traditional computer programs, which are written by software engineers who specify the step-by-step computations of transforming input data (called features) to output data (called labels). For example, to use the weight and height of a patient (features, or input data) to calculate the body mass index (BMI; a label, or output), a computer program can be written to perform the known calculation of BMI = weight/height (Clegg et al., 2013). In supervised ML, rather than providing the formula, the programmer simply gives these algorithms examples of patients with known weights, heights, and BMIs, and specific algorithms designed to learn from examples are used to build an ML model that predicts the BMIs for combinations of height and weight that were never seen in the initial set of examples provided. While ML would be a poor choice to determine BMI calculations from the weight and height since the relationship is known

ahead of time, it can be useful when the association is hard or impossible to specify by hand, such as using a digital picture of a person (features) to classify his or her BMI (label).

In medicine, ML models have been used to automate analysis of medical images, such as using eye fundus images (features) to diagnose diabetic retinopathy (labels; Gulshan et al., 2016) or using the sequence of data in a medical record (features) to predict patient outcomes, such as whether they are readmitted to the hospital (label; Rajkomar et al., 2018). Consented collection of digital data from patients during their daily life from wearable or ambient sensors can be used as input (features; Perez et al., 2019) for a variety of prediction tasks, such as onset of cognitive decline or worsened mobility (labels), which will be discussed in further detail below.

Input Data

Types of Sensors

The ubiquity of low-cost, miniature, and novel sensors allows for the collection of data that were previously too expensive or inconvenient to collect at scale. There is inconsistent terminology to categorize these sensors; some authors use the term “wearable” to emphasize the form factor and ease of collection, others use mobile health to highlight connection to a sensor carried in a mobile phone. However, data can be collected with sensors embedded in the environment (e.g., cameras or pressure sensors under a mattress to detect movement) that are similar to data collected with sensors worn on the body. This chapter considers the type of sensors that would detect data from daily living under proper consent regardless of whether they are wearable or ambient and refers to them as sensors despite the imprecision of this name.

Table 5-1 lists common sensors that are currently available commercially or in research devices that measure a host of signals, such as electrical signals (i.e., for electrocardiograms), acceleration/orientation (e.g., for movement), temperature, or audio (Heikenfeld et al., 2018; Mohr, Zhang, and Schueller, 2017; Ray et al., 2019). There are also a wide class of biosensors that use biological elements in the sensor itself (e.g., enzymes, cell receptors) that can be measured from the eye, mouth, skin, and more, although these are generally not commercially available and will not be discussed at length in this manuscript (Kim et al., 2019).

For ML, a key point is that the sensor data produce a raw signal that often undergoes further processing before outputting a human-understandable reading. For example, a photoplethysmographic sensor often outputs many readings of the heart rate that are averaged together in a process that produces a “final” reading periodically. The final reading is then fed into

TABLE 5-1 Selected Types of Sensors that Collect Data Outside Healthcare Settings

NOTE: These sensors can be used passively or actively, depending on the clinical application.

Active versus Passive Data Collection

Sensors commonly collect data passively, meaning a person is not actively engaging with the sensor as they go about their day (Sim, 2019).

an ML model. The details of this preprocessing done prior to the output of a visible sensor “reading” are idiosyncratic to a manufacturer, and these idiosyncracies are on top of the known issue that sensor data from the same type of device but different manufacturers are not equally accurate. Variations in sensor quality and sensor-data processing make validation and comparability of readings across all devices used in a study critical (Wang et al., 2017).

For example, simply carrying a smartphone is sufficient for accelerometers, barometers, and GPS sensors to track activity and movement. Passive sensing generates a sequence of measurements of variable duration and therefore length, and ML models specific to dealing with sequences exist to model this type of data.

Use of passive sensor data is likely more suitable for aging populations who may not wish to actively engage with devices, have difficulty using them, or are less comfortable performing active assessments themselves.

Sensors can also be intentionally engaged for active or functional assessment, such as performing a 6-minute walk test by carrying a phone; in this case, data collection would require the user to actively indicate the beginning and end of the test (even though the phone is also passively tracking movement as well). The active engagement might be triggered by a sensor reading, as when a user’s wristwatch sensor detects an arrhythmia and so prompts the user to report whether they are experiencing any symptoms of atrial fibrillation. Because active data collection like time exertion and electronic patient-reported outcomes is done under more controlled settings or with specific prompts than passive data collection throughout the day, the generated sensor data have less variation, and models can potentially be built with fewer data.

Outputs of a Model

A supervised ML model is trained to associate a sequence of sensor measurements with a specific output (i.e., label), and the output is intimately tied to the clinical purpose of the model. This section describes attributes of outputs from a machine learning and clinical research perspective.

Detection, Classification, and Prediction Machine Learning Outcomes

In traditional research, the output is called the primary outcome, and it is typically assessed at the end of a prespecified follow-up period. As shown in Figure 5-1, with sensor data, “output” can refer to several things. It can designate a measurement of the sensor data (detection); a secondary measurement made while sensor data are actively being collected (classification); or an outcome that will occur in the future (prediction).

Consider a wearable sensor that produces an electrocardiogram. A model could be used to detect if the recorded electrical pattern is consistent with atrial fibrillation. If the user is prompted to indicate their emotional state of anxiety at the time of an elevated heart rate, a model could use the same sensor to classify emotional state. A model could also predict if a patient, currently in sinus rhythm, will develop atrial fibrillation in the future (Attia et al., 2019).

While it is important that input data, whether passively or actively collected, be collected over representative populations, it is critical that labels, whether they are detections, classifications, or predictions, be of high quality compared to a reference standard. Reference standards themselves often require subjective clinical judgment, which may require multiple expert raters to reduce the intra- and inter-rater variability (Liu et al., 2019). Sensor data have the additional challenge of being extremely long, and annotating every segment of data may be infeasible; additional techniques may be necessary to coarsely tag parts of the sequence that require precise labeling (Yeung et al., 2019).

Detection and classification have numerous uses for aging populations, such as detecting abnormal vital signs or classifying activity (e.g., getting out of a chair) as indicative of frailty. Trends of classification, such as decreased activity or movement, can be used for direct clinical management (e.g., identification of worsening heart failure) or used as an interpretable feature and input of another ML model to predict admission to the hospital.

Prediction is critical to enable healthy aging because one of the most problematic expressions of aging is frailty, which has not been shown to be reversible (Clegg et al., 2013). Identifying patients who will become frail before they actually do is the critical first step to delaying or averting its onset. However, since frailty is a progressive clinical condition across a variety of age- and disease-related changes, even detection of the initial stages of frailty is a form of prediction, highlighting the related nature of detection, classification, and prediction. However, as noted below, predicting the future does not mean it is possible to change it.

Clinically Applicable Outcomes

A commonly described label is “onset of a disease state,” so that patients and their clinicians can be alerted early of an impending condition and take preventive action. For example, a continuous glucose monitor might be used to predict onset of diabetes within 3 years. Related extensions include detecting or predicting worsening of disease (e.g., automatic monitoring of daily tremor activity in patients with parkinsonism or prediction of manic episodes) and identifying patients who have specific subtypes of a disease and so may have a different expected trajectory or respond to different management.

For all outcomes, it is critical to distinguish hard versus surrogate endpoints. Hard, or clinical, endpoints, like survival or clinically noticeable change of how patients feel or function, are of true interest to patients and investigators, although these labels may be difficult or time consuming to collect for large groups of patients. Surrogate outcomes are laboratory or sensor measurements that are thought to be correlated with hard endpoints, such as detection of atrial fibrillation, which is strongly associated with stroke. However, it is well known in clinical research that successful prediction of surrogate endpoints is not guaranteed to lead to better hard outcomes, and in many cases, it can lead to worse or unintended consequences (Mandl and Manrai, 2019; Prasad et al., 2015; Weintraub, Lüscher, and Pocock, 2015).

Cohort Selection as It Relates to Outcomes

ML research traditionally focuses on defining input features and output labels, but for clinical applications, the population of patients for whom data and outcomes are collected—referred to here as the cohort—is equally significant but doesn’t always register in the input features.

ML models are more accurate when trained on data with high proportions of positive labels; in clinical research this corresponds to the percentage of enrolled patients who meet the definition of the primary outcome. While that can be modulated by selection of the output of interest (e.g., detecting a commonly seen surrogate outcome versus predicting a rare hard endpoint), it is also affected by the patient population studied, or the cohort.

This effect is so pronounced that in clinical research, the cohort of enrolled patients determines the classification of the study itself. Consider building a model to predict the increase of a patient’s hemoglobin A1c (label), a marker of diabetes, using consented activity and heart rate monitoring. If healthy patients are enrolled, the model becomes a risk biomarker (risk of disease), but if the patients already have diabetes, it becomes a monitoring biomarker (monitoring of known disease), and if the patient is

on treatment, it becomes a pharmacodynamic response biomarker (predicting treatment response).

From an ML perspective, these differences do not affect how a model is constructed, trained, or evaluated. But there are significant clinical implications as to whether the model is appropriate to use for various clinical populations. In addition to the cohort’s effect on the rate of positive labels and clinical generalizability, the type of data collection itself may induce selection bias into the cohort. Patients who are willing to wear, charge, update, and maintain sensor equipment over long periods of time may not reflect the age or socioeconomic status of a population of interest (Hicks et al., 2019). In particular, aging populations may worry that they do not have the competence to operate technology, that abnormal readings may induce health anxiety, or that the technology may be used to displace in-person monitoring and care (Sanders et al., 2012). Therefore, understanding the cohort and possible sources of bias is a critical step before building any ML model, especially related to aging populations.

CLINICAL STUDY CONSIDERATIONS

There are often high-level objectives for using sensor data, such as promoting healthy lifestyles and healthy aging to avert the onset of preventable diseases and enable seniors to continue living independently at home. Yet achieving these goals with sensor data and ML requires considerations of the clinical study nuances in addition to enrolling large cohorts of patients, recording high-quality input data, and obtaining adjudicated outcomes (Mohr, Zhang, and Schueller, 2017).

What Is the Right Label?

Applying ML to clinical data gathered by sensors requires consented, discrete, measurable, and reproducible labels that may not always be possible or easy to obtain in widespread populations. Hard endpoints like cognitive decline or death may take decades to occur, and clinical outcomes, like diagnosis, require regular clinical assessments that are not uniformly rigorous or applied across a population. There is a tendency to use surrogate endpoints related to specific sensor measurements that are known to be correlated to health outcomes, such as blood pressure or glucose levels. It is assumed that accurate detection or prediction of these metrics will lead to better health, especially if the metrics are related to modifiable factors (e.g., exercise or better diet). However, there are multiple examples in healthcare where successful interventions to achieve surrogate outcomes of reduced arrhythmia burden, hypertension, and hyperglycemia, led to worse patient outcomes, as shown below in Table 5-2.

TABLE 5-2 Case Studies Where Surrogate Outcomes Were Misleading

Are Relevant Data Collected Based on the Understanding of the Prediction Task?

Sensor data are modified by a host of factors that affect readings and measurements in nonobvious ways. For example, a newly physically active individual may develop a slower heart rate due to improved cardiovascular health, or the same finding may reflect that he is newly employed and now has health insurance to pay for a prescribed beta-blocker for migraine prevention. Traditional clinical studies have protocols to try to discern plausible causal factors that account for changes in outcomes. Because ML models may discern patterns not apparent to humans, if these alternative factors are not collected and analyzed, the model may produce spurious or misleading predictions. Clinical research expertise that focuses on a broad understanding of the phenomenon studied—not purely the technical details of the sensor or ML engineering—is necessary to combat this risk.

Will Producing a Model Actually Help?

The premise of using ML to analyze personal sensor data is that knowledge of what is detected, classified, or predicted will help an individual live a better life. It is often unclear if users change behaviors in response to recorded sensor data, or that users more likely to record sensor data in the first place will change their behavior (McConnell et al., 2018; Patel, Asch, and Volpp, 2015; Sperrin et al., 2016). In cases where the data induced intended behavior change, current evidence in mobile health studies shows only temporary, limited effectiveness for domains like improved activity (McConnell et al., 2018). Indeed, one study showed that use of wearable technology to assist in weight loss compared to traditional interventions led to less weight loss (Jakicic et al., 2016), highlighting the risk that sensor data may actually worsen outcomes through mechanisms that, in this case, even the investigators found unclear.

This is not a small concern that can be written off as inadequate hardware or software; it is a fundamental aspect of clinical experience that accurate detection and prediction do not necessarily correspond to better outcomes. For example, thyroid cancer screening programs in South Korea led to a rapid increase in detection of this cancer, and nearly all patients diagnosed were treated (Ahn, Kim, and Welch, 2014). Yet this treatment has not led to better hard outcomes (e.g., longer survival), and treated patients have experienced substantial complications from therapy; understanding the difference between underdetection and overdiagnosis is critical.

In an extreme case, the video game Pokémon Go successfully motivated increased physical activity but was sometimes followed by severe cases of trauma due to players’ inattention to their surroundings (Barbieri et al.,

2017). This is relevant to older adults because successful interventions to improve activity or other surrogate outcomes for elderly patients may concomitantly raise unanticipated risks, such as injuries that frail individuals may not recover well from.

Outcomes are also affected by constraints in the environment that are nonmodifiable, such as less activity due to living in a nonwalkable city (Sadik-Khan and Solomonow, 2017) or nonideal food choices associated with living in a food desert (Kelli et al., 2019). In these cases, policy or environmental changes may be more important interventions than personalized models.

How Predictive Is Sensor Data?

Is a continuous stream of sensor data required for an ML task? Although the premise of sensors is that daily habits and physical activity can substantially alter clinical outcomes, the experience from clinical trials shows that many drugs designed to induce a physiological effect actually have only modest treatment effects (Califf and DeMets, 2002). If lifestyle habits are thought of as inducing potential physiological changes related to health outcomes, then discerning the effect of each habit, especially when multiple habits occur sequentially in various orders and combinations, is extraordinarily difficult (Gottesman et al., 2019).

Prediction using continuous, consented measurement may not be more accurate than traditional episodic data collection or may not have incremental performance worth the burden of additional collection (Insel, 2017). Moreover, if new medical therapies or environmental changes are introduced, predictions using data from past patients may become stale or inaccurate.

What Are the Effects of Healthcare Disparities in Data and Machine Learning?

Collecting and using consented data from groups that have experienced discrimination or human and structural biases brings the attendant risk of worsening healthcare disparities (Rajkomar et al., 2018). It is known that healthcare outcomes are affected by social determinants of health, education, the criminal justice system, and more (Zimmerman and Anderson, 2019). The hope of using sensor data is that physiological or physical activity might be used directly to forecast health, but it is impossible to disentangle the effect of physical activity from all the other factors, especially in the face of ML. The net effect is that investigators need to carefully consider the interplay of healthcare disparities, collection of data, and creation of labels.

The complexity of ML models can create a pervasive influence of disparities that requires vigilance to detect. ML models can identify signals in the data that cannot be identified by humans (Poplin et al., 2018), and the imprints of the social determinants of health are subtly imprinted on all types of data. For example, consider a wearable sensor that measures a sequence of heart rates. To a human, the sequences from a device from one manufacturer might look the same as one from another manufacturer, but the idiosyncratic processing of the raw data can leave signatures in the data that are invisible to the human eye but distinctly present. This means that sensors that purport to measure the same physiological attribute may generate sequences that reveal as much information about the device itself as the heart rate of the patient; a model could therefore possibly distinguish data from “expensive” versus “inexpensive” sensors and use a derived socioeconomic indicator of wealth rather than the trends of the values themselves for prediction. This requires clinical and research expertise to know what to look for, and it requires data science expertise to identify and potentially address the effect of healthcare disparities on the results (Rajkomar et al., 2018).

In aging populations, there is especially the risk of privileged bias, agency bias, and informed mistrust. Privileged bias refers to the phenomenon of aging populations not having a voice in the types of technologies being developed that they can use or afford. As a result of privileged bias, systems may not be designed to solve the problems facing aging populations, such as limited internet connectivity or e-literacy that limits adoption of even interested elderly patients (Van Winkle, Carpenter, and Moscucci, 2017). Agency bias indicates a situation in which stakeholders do not have input into types of problems that they want solved. For example, aging populations may not be included in the decision-making process of building and deploying the models. Informed mistrust describes a situation where stakeholders do not trust the systems built to help them. This might happen, for example, when researchers may be financially incentivized to solve problems faced disproportionately by the well educated and wealthy, introducing possibly warranted skepticism that the models are generalizable. These problems are compounded by sources of bias in the data (e.g., nonrepresentative patients being enrolled) and the prediction of surrogate outcomes (Obermeyer et al., 2019).

There is no single solution to solve all of these problems, but there are recommendations on best practices to be upheld during all phases of developing ML models, including design, data collection, training, evaluation, launch review, and postdeployment (Rajkomar et al., 2018).

FUTURE WORK

Investigators face significant challenges in study design, data collection, and ML-based analysis. What are some paths forward?

Large-scale studies (All of Us Research Program Investigators, 2019) studying aging populations over long time periods will likely be a critical source of new insights. Existing studies have shown the feasibility of enrolling large numbers of patients in a short time period (Perez et al., 2019), but obtaining verifiable longitudinal data on those participants remains challenging both for logistical reasons and for lack of interoperability (Rajkomar, Dean, and Kohane, 2019). Applying commercially available sensors and tracking clinically relevant hard outcomes will likely promote better forecasting of future health events and deterioration, but the full cycle of trial development, analysis, and validation of these efforts may be protracted.

However, many relevant health outcomes are largely specific to older adults, such as the onset of frailty or progression of Parkinson disease. Studies that enroll patients at higher risk for these outcomes might be less generalizable to a wide population but can still provide insight for vulnerable patients. Although using hard outcomes in large-scale studies is preferable, thoughtfully using surrogate outcomes in smaller-scale but high-risk cohorts can accelerate knowledge generation and direct limited resources to run larger, expensive trials with hard outcomes. The rapid development of new wearables means that the ability to rapidly evaluate sensors for clinical promise is increasingly important if researchers are to design studies that take advantage of new technologies (Kim et al., 2019). A key insight is that identifying the specific clinical challenges, including the relevant cohorts and outcomes, requires traditional clinical research experience; such selection requires clinical researchers working alongside engineers and ML experts.

Future work will need to consider the significant additional challenges beyond detection, classification, and prediction. The critical challenge to improve the process of aging and promoting health will be finding interventions that can ameliorate problems if they are caught in real time or in advance (Kourtis et al., 2019).

CONCLUSIONS

Sensor data collected, with consent, from daily life promises to provide a peek at factors that lie beyond the measurement capabilities of traditional clinical studies that might affect health and aging. However, it is known that health is determined by many factors, some within individual control but many outside of it, including policy, social determinants, physical and

environmental determinants, biology, and access to health services (Determinants of Health, 2020). Moreover, while this chapter focused on key scientific challenges, there are a plethora of other key issues of regulatory, data security, privacy, workflow, interoperability, ethical, and legal considerations (Izmailova, Wagner, and Perakslis, 2018).

There should be optimism that new technology will deepen our understanding of health and aging, but clinical experience cautions that the path will be difficult and full of dead ends. It will require thoughtful application of best practices in sensor design, ML, and clinical research to yield useful and generalizable knowledge that helps older patients.

REFERENCES

Fisher, C.K., Smith, A.M., Walsh, J.R., Coalition Against Major Diseases, & Abbott, Alliance for Aging Research, Alzheimer’s Association, Alzheimer’s Foundation of America, AstraZeneca Pharmaceuticals LP, Bristol-Myers Squibb Company, Critical Path Institute, CHDI Foundation, Inc., Eli Lilly and Company, F. Hoffmann-La Roche Ltd, Forest Research Institute, Genentech, Inc., GlaxoSmithKline, Johnson & Johnson, National Health Council, Novartis Pharmaceuticals Corporation, Parkinson’s Action Network, Parkinson’s Disease Foundation, Pfizer, Inc., sanofi-aventis. Collaborating Organizations: Clinical Data Interchange Standards Consortium (CDISC), Ephibian, Metrum Institute. (2019). Machine learning for comprehensive forecasting of Alzheimer’s Disease progression. Scientific Reports, 9(1), article13622.

Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L.A. (2019). Guidelines for reinforcement learning in healthcare. Nature Medicine, 25(1), 16–18.

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., & Webster, D.R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22), 2402–2410.

Heikenfeld, J., Jajack, A., Rogers, J., Gutruf, P., Tian, L., Pan, T., Li, R., Khine, M., Kim, J., Wang, J., and Kim, J. (2018). Wearable sensors: Modalities, challenges, and prospects. Lab on a Chip, 18(2), 217–248.

Hicks, J.L., Althoff, T., Sosic, R., Kuhar, P., Bostjancic, B., King, A.C., Leskovec, J., & Delp, S.L. (2019). Best practices for analyzing large-scale health data from wearables and smartphone apps. NPJ Digital Medicine, 2, 45.

Insel, T.R. Digital phenotyping: Technology for a new science of behavior. (2017). JAMA, 318(13), 1215–1216.

Izmailova, E.S., Wagner, J.A., & Perakslis, E.D. (2018). Wearable devices in clinical trials: Hype and hypothesis. Clinical Pharmacology & Theraputics, 104(1), 42–52.

Jakicic, J.M., Davis, K.K., Rogers R.J., King, W.C., Marcus, M.D., Helsel, D., Rickman, A.D., Washed, A.S., & Belle, S.H. (2016). Effect of wearable technology combined with a lifestyle intervention on long-term weight loss: The IDEA randomized clinical trial. JAMA, 316(11), 1161–1171.

Kavanagh, B.P., & McCowen, K.C. (2010). Clinical practice. Glycemic control in the ICU. New England Journal of Medicine, 363(26), 2540–2546.

Kelli, H.M., Kim, J.H., Samman Tahhan, A., Liu, C., Ko, Y.A., Hammadah, M., Sullivan, S., Sandesara, P., Alkhoder, A.A., Choudhary, F.K., Gafeer, M.M., Patel, K., Qadir, S., Lewis, T.T., Vaccarino, V., Sperling, L.S., & Quyyumi, A.A. (2019). Living in food deserts and adverse cardiovascular outcomes in patients with cardiovascular disease. Journal of the American Heart Association, 8(4), e010694.

Kim, J., Campbell, A.S., de Ávila B.E-F., & Wang, J. (2019). Wearable biosensors for healthcare monitoring. Nature Biotechnology, 37(4), 389–406.

Kourtis, L.C., Regele, O.B., Wright, J.M., & Jones, G.B. (2019). Digital biomarkers for Alzheimer’s disease: The mobile/wearable devices opportunity. NPJ Digital Medicine, 2, article 9. doi:10.1038/s41746-019-0084-2.

Liu, H., Bravata, D.M., Olkin, I., Nayak, S., Roberts, B., Garber, A.m, & Hoffman, A.R. (2007). Systematic review: The safety and efficacy of growth hormone in the healthy elderly. Annals of Internal Medicine, 146(2), 104–115.

Liu, Y., Chen, P-H.C., Krause, J., & Peng, L. (2019). How to read articles that use machine learning: Users’ guides to the medical literature. JAMA, 322(18), 1806–1816.

Mandl, K.D., & Manrai, A.K. (2019). Potential excessive testing at scale: Biomarkers, genomics, and machine learning. JAMA, 321(8), 739–740.

McConnell, M.V., Turakhia, M.P., Harrington, R.A., King, A.C., & Ashley, E.A. (2018). Mobile health advances in physical activity, fitness, and atrial fibrillation: Moving hearts. Journal of American College of Cardiology, 71(23), 2691–2701.

Mohr, D.C., Zhang, M., & Schueller, S.M. (2017). Personal sensing: Understanding mental health using ubiquitous sensors and machine learning. Annual Review of Clinical Psychology, 13, 23–47.

Moyé, L.A., & Tita, A.T.N. (2002). Defending the rationale for the two-tailed test in clinical research. Circulation, 105(25), 3062–3065.

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.

Patel, M.S., Asch, D.A., & Volpp, K.G. (2015). Wearable devices as facilitators, not drivers, of health behavior change. JAMA, 313(5), 459–460.

Perez, M.V., Mahaffey, K.W., Hedlin, H., Rumsfeld, J.S., Garcia, A., Ferris, T., Balasubramanian, V., Russo, A.M., Rajmane, A., Cheung, L., Hung, G., Lee, J., Kowey, P., Talati, N., Nag, D., Gummidipundi, S.E., Beatty, A., Hills, M.T., Desai, S., Granger, C.B., Desai, M., & Turakhia, M.P.. (2019). Large-scale assessment of a smartwatch to identify atrial fibrillation. New England Journal of Medicine, 381(20), 1909–1917.

Pfeffer, M.A., & McMurray, J.J.V. (2016). Lessons in uncertainty and humility—Clinical trials involving hypertension. New England Journal of Medicine, 375(18), 1756–1766.

Poplin, R., Varadarajan, A.V., Blumer, K., Liu, Y., McConnell, M.V., Corrado, G.S., Peng, L., & Webster, D.R. (2018). Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nature Biomedical Engineering, 2(3), 158–164.

Prakash, S.K.A., & Tucker, C.S. (2018). Bounded Kalman filter method for motion-robust, non-contact heart rate estimation. Biomedical Optics Express, 9(2), 873–897.

Prasad, V., Kim, C., Burotto, M., & Vandross, A. (2015). The strength of association between surrogate end points and survival in oncology: A systematic review of trial-level meta-analyses. JAMA Internal Medicine, 175(8), 1389–1398.

Rajkomar, A., Dean, J., & Kohane, I. (2019). Machine learning in medicine. New England Journal of Medicine, 380(14), 1347–1358.

Rajkomar, A., Oren, E., Chen, K., Dai, A.M., Hajaj, N., Hardt, M., Liu, P.J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G.E., Irvine, J., Le, Q., Litsch, K., Mossin, A., Tansuwan, J., Wang, D., Wexler, J., Wilson, J., Ludwig, D., Volchenboum, S.L, Chou, K., Pearson, M., Madabushi, S., Shah, N.H., Butte, A.J., Howell, M.D., Cui, C., Corrado, G.S, & Dean, J. (2016). Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1), article 18.

Rajkomar, A., Hardt, M., Howell, M.D., Corrado, G., & Chin, M.H. (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine, 169(12), 866–872. doi:10.7326/M18-1990.

Ray, T.R., Choi, J., Bandodkar, A.J., Krishnan, S., Gutruf, P., Tian, L., Ghaffari, R., & Rogers, J.A. (2019). Bio-integrated wearable systems: A comprehensive review. Chemical Reviews, 119(8), 5461–5533.

Sadik-Khan, J., & Solomonow, S. (2017). Improving public health by making cities friendly to walking and biking: Safer, more active transportation starts with the street. JAMA Internal Medicine, 177(5), 613–614.

Sanders, C., Rogers, A., Bowen, R., Bower, R., Hirani, S., Cartwright, M., Fitzpatrick, R., Knapp, M., Barlow, J., Hendy, J., Chrysanthaki, T., Bardsley, M., & Newman, S.P. (2012). Exploring barriers to participation and adoption of telehealth and telecare within the Whole System Demonstrator trial: A qualitative study. BMC Health Services Research, 12, article 220.

Sim, I. Mobile devices and health. (2019). New England Journal of Medicine, 381(10), 956–968.

Sperrin, M., Rushton, H., Dixon, W.G., Normand, A., Villard, J., Chieh, A., & Buchan, I. (2016). Who self-weighs and what do they gain from it? A retrospective comparison between smart scale users and the general population in England. Journal of Medical Internet Research, 18(1), e17.

Taler, S.J. (2018). Initial treatment of hypertension. New England Journal of Medicine, 378(7), 636–644.

Van Winkle, B., Carpenter, N., & Moscucci, M. (2017). Why aren’t our digital solutions working for everyone? AMA Journal of Ethics, 19(11), 1116–1124.

Wang, R., Blackburn, G., Desai, M., Phelan, D., Gillinov, L., Houghtaling, P., & Gillinov, M. (2017). Accuracy of wrist-worn heart rate monitors. JAMA Cardiology, 2(1), 104–106.

Weintraub, W.S., Lüscher, T.F., & Pocock, S. (2015). The perils of surrogate endpoints. European Heart Journal, 36(33), 2212–2218.

Yeung, S., Rinaldo, F., Jopling J., Liu, B., Mehra, R., Downing, N.L., Guo, M., Bianconi, G.M., Alahi, A., Lee, J., Campbell, B., Deru, K., Beninati, W., Fei-Fei, L., & Milstein, A. (2019). A computer vision system for deep learning-based detection of patient mobilization activities in the ICU. NPJ Digital Medicine, 2, article 11.

Zimmerman, F.J., & Anderson, N.W. (2019). Trends in health equity in the United States by race/ethnicity, sex, and income, 1993–2017. JAMA Network Open, 2(6), e196386.

__________________