National Academies Press: OpenBook

Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series (2019)

Chapter: 8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?

« Previous: 7 How Tightly Should Investigators Attempt to Control or Restrict Treatment Quality in a Pragmatic or Real-World Trial?
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 91
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 92
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 93
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 94
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 95
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 96
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 97
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 98
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 99
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 100
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 101
Suggested Citation:"8 Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.
×
Page 102

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

8 Obscuring Intervention Allocation in Trials to Generate Real- World Evidence: Why, Who, and How? Key Messages Identified by Individual Workshop Participants Obscuring intervention allocation, or blinding, allows researchers to study the effects of an intervention without influence from patient and provider perceptions. However, it may not always be appropriate or feasible. (London, Watanabe) The appropriateness of blinding is highly dependent on the context of the study, the research question at hand, and the specific uncertainties for that study. (Critchlow, Dreyer, London, Smith, Vardeny) Uncertainties in a study can be classified along two axes: ensemble efficacy and utilization factors. The interaction of these two for a particular study can indicate whether blinding is appropriate. (London) Blinding can impart risks to the study, such as patient exposure to either sham procedures or to incomplete care from their provider. These and other risks are balanced against the benefits of blinding in order to make a decision. (Dreyer, Hernandez, London, Simon) Decisions on blinding can also be influenced by practical considerations, such as effects on study cost, feasibility of masking or delivering the treatments, patient preferences, and the generalizability of data from blinded study sites. (Dreyer, Reynolds, Smith) Patient and provider bias, and how it may influence a study, can be difficult to predict in advance. Bias may not affect some outcomes, such as quantitative lab readings or all- cause mortality, but can affect other more subjective outcomes or have other effects such as ascertainment or treatment bias. (Critchlow, Dreyer, Graham, Smith) P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS 91

92 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT Obscuring intervention allocation—commonly known as blinding—is used to reduce the likelihood that those involved in the trial will be influenced by the treatment assignment, said James P. Smith, deputy director of the Division of Metabolism and Endocrinology Products at the U.S. Food and Drug Administration’s Center for Drug Evaluation and Research. If those involved are influenced by treatment assignment, it may affect the outcome of the trial. Blinding allows researchers to study the effects of the intervention without the influence of patient and provider perceptions. However, blinding is not always appropriate or feasible, said Alex John London. Trial design features such as blinding have to be justified by the contribution they make to the evidence quality and the relative risks and costs compared with alternative designs, he said. This session in the third workshop examined the topic of blinding, and session moderator Jonathan Watanabe, associate professor of clinical pharmacy and National Academy of Medicine Anniversary Fellow in Pharmacy, University of California, San Diego, asked participants to consider: How might variability in knowledge of treatment group assignment affect provider and patient adherence and outcomes? How might variability in knowledge of treatment assignment affect study cost and reliability? What key factors could affect decisions to obscure intervention allocation? This topic was highlighted by some participants at the second workshop as an area that needed further exploration, so a session was held on the topic at the third workshop. ILLUSTRATIVE EXAMPLES To explore the issues surrounding obscuring treatment intervention and blinding, speakers at the second and third workshops presented case studies as illustrative examples of the considerations that go into designing and conducting a real-world study. INVESTED Trial Orly Vardeny, associate professor of medicine at the University of Minnesota and of the Minneapolis VA [U.S. Department of Veterans Affairs] Center for Chronic Disease Outcomes Research, told workshop participants about the INVESTED [INfluenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated heart failure] trial. Influenza leads to significant morbidity and mortality, said Vardeny, and there have been several analyses that documented a temporal association between influenza infection and cardiovascular events(Madjid et al., 2007; Thompson et al., 2003, 2004). Researchers have sought to further understand how the influenza vaccination could affect cardiovascular events. The first step, said Vardeny, was a meta-analysis of six studies that demonstrated that influenza vaccination could reduce cardiovascular events (Udell et al., 2013). The next research question, said Vardeny, was to determine if patients with heart failure exhibited the same immune response to the flu vaccine. This analysis showed that in fact, heart failure patients had a reduced antibody response compared with patients without heart failure (Vardeny et al., 2009). Researchers were able to increase the antibody response by giving a higher dose of the flu vaccine, she said (Van Erman et al., 2013). P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE RWE 93 Based on this initial research, the INVESTED 1 researchers set out to examine whether the increased immune response from the high-dose vaccine would translate into better cardiovascular outcomes (see Figure 8-1). The researchers designed a randomized trial, comparing high-dose trivalent influenza vaccine to standard-dose quadrivalent influenza vaccine. Vardeny noted the standard-dose quadrivalent vaccine for the active control group was chosen because it would be unethical to have an unvaccinated control group, and the quadrivalent vaccine is the current standard of care. Researchers have enrolled approximately 3,000 participants out of a target of 9,300 participants for this pragmatic study so far, and have contacted them up to 4 times per year by phone to ascertain endpoints. The primary endpoint is a composite of death or cardiopulmonary hospitalization. FIGURE 8-1 Study design for the INVESTED trial. SOURCE: Vardeny presentation, July 17, 2018. Both providers and participants are blinded in the study, which is accomplished by a third party vendor affixing identical labels over the individual-dose syringes. Vardeny said that because there were some ways to tell the two vaccines apart (e.g., the high dose causes more pain during injection), the study was designed so that the person who administers the vaccine is not the same person who conducts the ascertainment phone calls later in the season. Vardeny said the choice to conduct a double-blind study was based on a few considerations (see Box 8-1). There were reasons not to blind, she said. Blinding delays the study by several weeks while the third party vendor blinds and distributes the vaccines. During this period, the flu vaccine becomes available to the general public, which reduces the number of 1 See https://clinicaltrials.gov/ct2/show/NCT02787044 (accessed January 4, 2019). P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

94 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT people eligible to participate in the trial because they already received the vaccine. However, said Vardeny, the reasons for blinding outweighed these considerations. There are perceived differences in efficacy of these vaccines, she said. For example, the high-dose vaccine is only approved for individuals aged 65 and over. Although there is no recommendation that the standard of care for people over 65 should be the high-dose vaccine, some providers or patients may perceive that all people over 65 should get the high-dose vaccine. Without double blinding, these perceptions may have resulted in systematic bias in terms of patients presenting at the hospital, or providers urging hospitalization. BOX 8-1 Other Examples of Blinded and Non-Blinded Studies Discussed Throughout the Workshop Series as Presented by Individual Workshop Participants The Salford Lung Studies to understand the effectiveness of asthma and chronic obstructive pulmonary disease drugs in the usual care setting. discussed in detail by Gibson and in Chapter 3. Graham discussed in this session that patients were randomized to either group, and the study was conducted open label (i.e., not blinded). There were several reasons why the researchers chose not to blind patients in the study, said Graham. First, in the United Kingdom (where the study was conducted), the standard practiceis for patients to receive their medicine from a pharmacist who educates them on the use of the drug and the importance of adherence. Had the study been blinded, this real-world aspect of clinical care would have been altered and potentially caused confusion or changes in adherence for patients. In addition, said Graham, the delivery mechanism of inhalers can impact outcomes, aside from the specific drug that is inhaled. Blinding may have complicated the ability to see the benefit of the inhaler itself, he said. Novel oral anticoagulants (NOACs) versus warfarin for controlling blood pressure. Hernandez presented trials comparing NOACs with warfarin; three were double-blind trials (see Chapter 6 for full details). He noted that randomizing more than 14,000 patients to two treatment arms in a double-blind study was “quite a challenge.” He also noted that any study that involves warfarin is challenging to maintain the blind because warfarin requires frequent monitoring. Researchers must also ensure that patients in the warfarin arm are appropriately anticoagulated, without revealing the treatment assignment. Lithium for suicide prevention. Ira Katz presented a research project that evaluated lithium for suicide prevention (see this chapter for full details). The trial was double-blinded and placebo-controlled. The INTERSEPT and PRIDE trials. Larry Alphs presented two pragmatic trials (see this chapter for full details). The INTERSEPT trial tested two oral antipsychotic drugs, and was not blinded because of differences in known side effect profiles, additional blood draw monitoring required by only one of the drugs, and because the investigators judged that it was unethical for this study to be blinded because that may have inhibited individual providers’ ability to give the best care to their patients. The PRIDE trial compared an oral to a long-acting injectable antipsychotic and was open label because the different delivery mechanisms would have been impractical to blind. P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE RWE 95 DECISION AID The general issues discussed by individual workshop participants in the first and second workshops were used to develop a decision aid for the third workshop (see Figure 8-2). As with the other decision aids, the intention was to outline some questions to consider in order to make thoughtful choices in real-world evidence (RWE) study design. Participants at the third workshop reflected on these questions and offered feedback on the decision aid throughout the course of their discussions. In addition to offering specific feedback on the decision aid (see Box 8-2), workshop participants also discussed the difficulty of answering these types of questions prior to conducting a study. Simon summarized that although the questions in the decision aid may be good questions to consider, making decisions about the risks and benefits of blinding in a study a priori is enormously difficult. Watanabe asked the participants if they thought that the answers to the questions on the decision aid could be quantified a priori—for example, if researchers could, before a study, quantify the risks and benefits of blinding and make a decision based on these numbers. Cathy Critchlow, vice president at the Center for Observational Research at Amgen, John Graham, and Vardeny all said that quantifying would be enormously difficult. Critchlow added that this sort of quantification could be risky because it could make the researchers “feel better about making a decision that may not be based on something real.” Richard Platt wondered if some of these a priori decisions could be based on empirical data on bias from previous research—for example, data on how aggressively vaccinated patients are treated versus non-vaccinated patients. Vardeny said vaccinated patients are so different from unvaccinated patients that it would be difficult to isolate the effect of any provider bias. Smith said an alternative approach could be statistically adjusting for the bias after the research was complete, which would still require a judgment, but may be conceptually easier than quantifying the bias a priori. Simon suggested that there may be value, after a study is completed, in analyzing decisions made in the study to learn from any mistakes that may have been made. Smith concurred and said that “an autopsy of trials that do not go the way one expected can only be an incredibly valuable exercise,” and could be a best practice. Graham added that trial “autopsies” could be done not just for trials that gave a surprising result, but for all trials. He said that researchers could spend more time looking back at their decisions and the impact the decisions had on the results, and could include this information in their published results. Each study can give researchers insight into how to improve the next study, he said. P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

96 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT O BSCURING I NTERVENTION A LLOCATION IN T RIALS TO G ENERATE R EAL -W ORLD E VIDENCE : W HY , W HO , AND H OW ? What expectations or preferences providers and patients be expected to have regarding benefits and adverse effects of study interventions? How might those preferences or expectations influence: Intervention uptake or adherence? Fidelity or intensity with which an intervention is delivered? Likelihood that beneficial or adverse effects would be reported or observed? How might those expectations or preferences differ in the settings where trial results will eventually be applied? How might concealing treatment allocation from patients and/or providers reduce biases due to preferences or expectations? How might concealing treatment allocation from patients or providers obscure meaningful differences between interventions? How might procedures necessary to conceal treatment allocation from patients and/or providers impact: The acceptability of trial participation to patients and or providers? The cost of the trial? The risk/benefit ratio of trial participation? P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE RWE 97 FIGURE 8-2 Decision aid on questions to consider regarding when and whether to obscure intervention allocation (commonly known as blinding) in trials intended to generate real-world evidence. NOTE: This decision aid was drafted by some individual workshop participants based on the discussions of individual workshop participants at the first and second workshops in the real-world evidence series. The questions raised are those of the individual participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine, and the figure should not be construed as reflecting any group consensus. SOURCE: Watanabe presentation, July 17, 2018. BOX 8-2 Feedback on the Decision Aid as Discussed by Individual Workshop Participants Several workshop participants offered specific feedback on the decision aid “Obscuring Intervention Allocation in Trials to Generate Real-World Evidence: Why, Who, and How?” The evaluation of a potential real-world trial needs to consider the existing knowledge base and what other studies have been conducted. For example, the Salford Lung Studies were planned after two standard, randomized, placebo-controlled trials had been conducted. This is a very different evaluation from using the Salford Studies for regulatory approval. The decision aid should reflect this contextual difference. (London) One of the questions on the decision aid asks, “How might concealing treatment allocations reduce bias?” This question should instead ask, “How much might concealing treatment allocations reduce bias?” (Dreyer) These questions will not generally have yes/no answers; rather the goal should be to estimate how much a research design decision (e.g., blinding) will affect bias or other issues. (Graham) DISCUSSION Context of the Decision Critchlow echoed an earlier discussion (see Chapter 6) in which participants stressed the importance of considering the context of the decision when designing a study that uses real- world data (RWD). The decision about whether or not to blind participants in a study, she said, should consider what the study is meant to accomplish: Is it to answer a question about effectiveness, is it to inform treatment guidelines, is it to get approval for a new treatment, or is it to expand the label indications for an already approved treatment? Depending on the use of the study, blinding may be more or less appropriate. Researchers, she said, need to ask themselves: Is blinding helping us to answer the right question and make the right decision? Nancy Dreyer, chief scientific officer at IQVIA, agreed with this framing, and added a more specific question: What is the expected effect estimate, and how likely is it to be missed? For example, if a study that will use RWD is about a chronic disease for which the treatments might make a small but important difference, blinding might be necessary to get the needed effect size for the decision to be made, she said. Smith agreed that making a decision about blinding depends on the research P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

98 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT question. For example, he said, if the research is seeking answers about whether patients are more likely to adhere to an injectable therapy or to an oral daily tablet, a blinded study cannot answer that question. London concurred with the other speakers, saying that trial design decisions should be based on the specific uncertainties that a study is aimed at mitigating (see Figure 8-3). FIGURE 8-3 Considerations around uncertainty in study design. SOURCES: London presentation, July 17, 2018; concept from London, 2018. For example, if there are uncertainties about efficacy, blinding may be necessary, whereas if there are uncertainties about effectiveness in the real world, blinding would not be warranted. London explained that real-world effectiveness is a function of two variables: ensemble efficacy and utilization factors. The term “ensemble efficacy” acknowledges the fact that a drug is not efficacious on its own—the therapeutic effect depends on the proper dose and schedule, the right population, and co-interventions or diagnostic requirements. Utilization factors include patient and provider awareness and preferences, cost, adherence, clinical capacity, and tolerability. Even if an ensemble is efficacious, said London, the real-world effectiveness of the drug depends on the utilization factors. For example, a drug could be efficacious for reducing blood pressure, but if it is unaffordable, difficult to adhere to, or poorly tolerated, it will not be effective in the real world because patients will not use it. Blinding may be more or less appropriate, depending on the state of knowledge about the ensemble efficacy and utilization factors, said London. If two interventions are being compared for real-world effectiveness, and it is known that Intervention 1 is more efficacious than Intervention 2, but the utilization factors are unknown, blinding would not be warranted (see Figure 8-4). The research question in this scenario, said London, is how these real-world factors P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE RWE 99 such as patient preference will impact the effectiveness of an intervention. If patients and providers were blinded, the utilization factors would remain unknown and thus the real-world effectiveness would not be understood. FIGURE 8-4 Blinding not warranted when utilization factors are unknown. NOTE: OIA = obscuring intervention allocation. SOURCE: London presentation, July 17, 2018; concepts from Kimmelman and London, 2015 and Moseley et al., 2002. However, he said, a measure of real-world effectiveness may not always be sufficient for making policy or care decisions, compared to a measure of efficacy. London gave the example of arthroscopic surgery for osteoarthritis of the knee. This procedure, he said, was performed about 650,000 times annually at a cost of $3.25 billion. However, when an RCT was performed to compare the procedure to a sham procedure, it was found to have no benefit (Moseley et al., 2002). The sham procedure, said London, was “largely theater” (manipulations, surface cuts, and bandages), but was necessary to obscure the intervention allocation. In this case, blinding was essential for a clear picture of intervention efficacy, and these efficacy data were necessary to determine the clinical merit of the procedure. Risks of Blinding When blinding is necessary to fully understand the effectiveness of an intervention, participants in the control group receive some sort of placebo drug or intervention, London said. This could range from a sugar pill, to a sham surgery that is “largely theater,” to invasive surgical procedures such as implanting cannulas in the brain to deliver saline in Parkinson’s trials, said London. There are risks to these procedures, and the benefits of a placebo are limited. The risks of these control procedures need to be disclosed to patients, he said. However, it is important to note that there are also risks involved with the active procedure, and there may also be no benefit. For example, while there are minor risks involved with a sham knee surgery, the real knee surgery is far more invasive and riskier. If the real knee surgery does not have any benefit, London said, the risks far outweigh the benefits. Both the risks from the placebo or sham procedure and the risks from receiving a potentially ineffective active intervention should be P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

100 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT communicated to participants, London said. (See Chapter 7 for more discussion of patient safety in real-world studies.) Another potential risk of blinding is that the participant population may be more homogeneous and more tightly controlled, which may make the results less generalizable, said Adrian Hernandez. The process of enrolling in a trial, and consenting to all of the processes necessary for blinding (e.g., complying with regular monitoring), may filter out some populations of participants. Blinding can “obscure the truth,” said Simon. “We fundamentally distort the nature of the treatment” by making two active treatments seem identical and by ensuring that the patient and provider experiences with the treatments are identical. For example, he said, if two treatments have very different dosing schedules, this difference may have an important bearing on the acceptability and adherence to those treatments, he said. While there are obvious advantages of blinding (e.g., mitigating bias), it can also constrain opportunities to learn about how a treatment will work in the real world. In discussions about whether or not to blind, said Dreyer, much attention is given to the risk of bias if a provider knows to which treatment group a patient belongs. However, she said, there are risks to the patient if a provider does not know which treatment he or she is receiving. For example, a provider may not know what adverse effects to monitor, or when a change in health status may indicate a significant problem. Dreyer said that while blinding can have benefits for the study, these benefits need to be weighed against the risks to the patient. Cost Considerations Researchers, said Dreyer, aim to get precise answers to research questions and then compare their answers to those that others get in order to confirm or refute the original hypothesis. However, Dreyer said that as a consumer and generator of data, researchers should be looking to see if two estimates are close, and certainly in the same direction, rather than expecting them to match exactly. Rather than designing a study to generate the most exact answer—which can increase costs significantly—she said, researchers should consider the big picture of the RWD’s intended use, and design studies accordingly. Dreyer said the economic value of different study designs could be considered in the designs; if the answer can be found with an open label, real-world study rather than from a more expensive randomized blinded trial—and it can be shown that bias is unlikely to explain the observed effect—it may be a better use of resources to choose the less expensive design, she said. Smith had a slightly different perspective. He said that while blinding can certainly increase the cost of a trial, an unblinded trial may “give you results that nobody believes,” which wastes time and resources. Practical Considerations For some interventions, such as vaccines, blinding is quite feasible and does not overly complicate the study, said Dreyer. However, for other interventions that involve treatments that are complex, sequential, or require monitoring and dose adjustment, it may be impractical or impossible to blind while keeping patient safety in mind. There are situations, said Smith, where blinding is simply not feasible. For example, some drugs have severe side effects, and to try to duplicate these side effects in the control group would be impossible or unethical. Blinding is a “tool that can be valuable in a narrow set of circumstances,” Dreyer said, but it should not necessarily be employed in all circumstances. P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

OBSCURING INTERVENTION ALLOCATION IN TRIALS TO GENERATE RWE 101 Hernandez noted that blinding could have the advantage of keeping participants in a trial. If a participant who does not know if he or she is getting the active treatment may be more likely to continue care, he said. Conducting a blinded study in an RWE may be difficult, said Rob Reynolds, Vice President and Global Health, Epidemiology, Pfizer. Blinded RCTs are difficult enough, he said, but in a real-world environment with diverse health care sites and patient populations, the feasibility challenges are even greater. Dreyer joked that “a lot of things are feasible if you spend enough money.” She added that when doing blinded studies in the real world, researchers generally perform the studies at academic centers that are more able to carry out this type of research. However, this may result in a lack of generalizability because these centers are often especially high-quality facilities with a high quality of care and follow practices that may not be routinely conducted in community settings. Patient Preferences One question on the decision aid asks, “How might procedures necessary to conceal treatment allocation from patients and/or providers impact the acceptability of trial participation to patients and/or providers?” Smith said the best way to answer this question is to ask the patients directly. Smith relayed an experience he had working with patients with a rare disease. He consistently heard that it would be impossible to conduct a placebo-controlled blinded trial in this population, but when meeting with a group of patients directly, the patients said if blinding “will give you the best data if we would be willing to go on a placebo for 6 months, we are happy to do so.” Smith said that assumptions about patients’ willingness to participate in blinded trials may not always be accurate. Critchlow added that in a context where there are fewer therapeutic options, patients might be more willing to take the risk of participating in a trial, whereas in a context with plenty of existing therapeutic options, the risk–benefit analysis of patients and providers is quite different. On another patient-related topic, John Burch, an investor with the Mid-America Angels Capital Network, asked whether participants in a blinded trial are ever able to get information about which treatment they received. “If patients are entitled to their own information,” he said, “does that include patient-specific data from research?” Graham said that from an ethical perspective, research is likely progressing to a point where patients will be able to get information about their treatment in a trial. Biases Blinding, said Smith, is one mechanism to reduce the possibility of bias that would affect the trial results. There are multiple types of bias that can be associated with treatment assignment, some of which are discussed below. Smith said bias is often subconscious, and it can be difficult to predict the direction and magnitude of the bias. Different patients or providers within the same trial might have biases that affect the results in different directions, he said. For example, if aware of treatment assignment, one patient may believe that a novel drug will help, and therefore may overstate its benefit. Another patient may believe that the standard of care is fine and could be skeptical of a company’s development of a novel drug. The direction and magnitude of various biases may differ depending on the specific disease area, type of intervention, or patient population. Overall, Smith said, “It is very difficult to predict how bias is going to creep in and what direction and magnitude it is going to be.” P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

102 EXAMINING THE IMPACT OF RWE ON MEDICAL PRODUCT DEVELOPMENT Patient Perception Without blinding, there is potential for bias due to patient perception, said Critchlow. For example, patient perception may impact the likelihood of complying with a treatment regimen as intended. Dreyer said that bias due to patient perception can differ, depending on the subjectivity of the measurements. For example, if the outcome of interest is hemoglobin A1C levels, patient perception is unlikely to have an effect. On the other hand, if the outcome of interest is pain level as reported by the patient, this could be influenced dramatically by perception, she said. Dreyer noted that for some outcomes, patient perception likely has no effect—“dead is dead.” Graham added that depending on the time line of the study, patient perception bias may wane over time. For example, if a patient gets a vaccine and feels ill 3 weeks later, does the patient’s initial idea of what treatment he received still affect his likelihood of seeking medical care? Graham said this is a very difficult area to have a black or white answer. Ascertainment Bias If the people who are assessing outcomes are not blinded, said Critchlow, it can result in ascertainment bias. The potential for ascertainment bias may depend on whether the endpoints are subjective or objective. For example, if the endpoint is a quantitative lab result, it is unlikely to be biased by the assessor knowing which group to which the patient is assigned. Outcomes assessors, said Smith, can be blinded to mitigate ascertainment bias, but it must be recognized that “they only assess what has been given to them.” For example, a blinded pathologist can read a biopsy specimen, with no knowledge of the clinical course or treatment. However, blinding the assessor does not automatically remove the potential introduction of bias. For example, although the pathologist may be blinded, the decision by the patient’s provider to conduct a biopsy could have been influenced by their knowledge of treatment assignment. In this scenario, said Smith, “the bias has already been introduced before the outcome got to the outcomes assessor.” Treatment Bias When providers are not blinded during a trial (i.e., double-blinded), said Smith, there is the worry that patients in one group are being treated differently than patients in the other group. For example, if a provider knows that a patient is on a placebo for suicide prevention versus an active treatment, the provider may be more likely to closely monitor the patient for signs of suicidality. The bias can also run in the other direction—if, for example, a provider knows that a patient is receiving an active but not well-understood treatment, the provider might monitor more closely for adverse events. Channeling Bias Channeling bias occurs later in the life cycle of a product when the adverse events associated with a product are well known, Jesse Berlin said during the second workshop. Patients who are at higher risk for these events are prescribed a different product; when these high-risk patients experience the event, the other product “starts to look like it’s increasing risk.” For example, non-steroidal anti-inflammatory drugs (NSAIDs) are associated with an increased risk of myocardial infarction (MI). Because of this, patients who are already at high risk of MI may be prescribed acetaminophen instead of NSAIDs, which results in a larger-than-expected group of acetaminophen patients experiencing MI. P REP UBLI CATI ON COP Y: UNCORR ECTED P ROOFS

Next: 9 Gaining Confidence in Observational Comparisons »
Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series Get This Book
×
Buy Paperback | $60.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Randomized controlled trials (RCTs) have traditionally served as the gold standard for generating evidence about medical interventions. However, RCTs have inherent limitations and may not reflect the use of medical products in the real world. Additionally, RCTs are expensive, time consuming, and cannot answer all questions about a product or intervention. Evidence generated from real-world use, such as real-world evidence (RWE) may provide valuable information, alongside RCTs, to inform medical product decision making.

To explore the potential for using RWE in medical product decision making, the National Academies of Sciences, Engineering, and Medicine planned a three-part workshop series. The series was designed to examine the current system of evidence generation and its limitations, to identify when and why RWE may be an appropriate type of evidence on which to base decisions, to learn from successful initiatives that have incorporated RWE, and to describe barriers that prevent RWE from being used to its full potential. This publication summarizes the discussions from the entire workshop series.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!