A useful procedure to precisely articulate a causal question is to describe a “target trial,” that is, a hypothetical randomized trial that would answer the question of interest if resource constraints or ethical issues did not preclude conducting it. The process of defining a target trial aids both in the definition of research questions and in the evaluation of various observational data sources and analysis strategies. To address the research question, the target trial is then emulated using available data sources.
The committee proposes two target trials that address the questions raised in the Statement of Task and then proposes a procedure to emulate those trials using existing Department of Veterans Affairs (VA) data. This chapter describes the research question the committee chose to address based on the Statement of Task and lays out the target trial framework. Chapter 3 follows by setting forth procedures for the use of observational data to emulate the target trials described in this chapter. For clarity, the committee uses the term “target trial” to refer to the hypothetical randomized trial that would directly address the research question; “protocol” to include all components of the design (e.g., patient enrollment, treatment strategies, outcome) and analysis of the target trial; and “observational analysis” to refer to the data analysis proposed to emulate the target trial using existing observational data.
The committee interpreted its task to focus on the following research question: What were the effects of opioid initiation and tapering1 strategies in the presence of benzodiazepines in veterans on all-cause mortality and suicide mortality from 2010 to 2017?2 The committee’s focus on this specific research question was a result of its interpretation of its task and the committee’s review of prior literature.
First, the committee determined that initiation and tapering represent two critical decision points in opioid treatment. A decision to initiate opioids is of necessity made for all people who eventually progress to long-term opioid therapy. Opioids can be initiated in multiple ways, including (1) patients with chronic pain are given an initial opioid prescription with the intent of treating them with long-term opioid therapy; (2) patients who have surgery, whether for a chronic pain condition or an unrelated condition, are given opioids post-operatively, and some of them stay on opioids long-term; and (3) patients are started on opioids for an injury or other non-surgical acute pain problem (e.g., acute low back pain, trauma), and some of them stay on opioids long-term. For simplicity of exposition, the committee decided to focus on the first group. However, all three groups contribute to the larger body of individuals who begin long-term opioid therapy and could be the focus of separate—and equally important—studies.
Tapering—broadly defined as any reduction in daily opioid dosage for a patient who is on long-term opioid therapy—is the second decision point of focus identified by the committee. Given the focus of the Statement of Task on adverse consequences, tapering is highly relevant for two main reasons: (1) the desire to avoid the known adverse consequences of long-term opioid therapy, such as increased mortality, is often a motivation for the decision to taper opioid dosage rather than continuing treatment, and (2) some clinicians and patients have raised concerns that, rather than reducing the risk for harm, tapering patients who are tolerant to opioids may contribute to adverse consequences, particularly suicide (discussed later in this chapter under “Potential Harms of Treatment”). The committee concluded that a study of the effect of opioid tapering on all-cause mortality and suicide would reduce clinical uncertainty and would be timely in the context of current opioid policy and practice decisions.
Second, the Statement of Task specifically focuses on “concomitant opioid and benzodiazepine” prescribing. After reviewing prior studies (see
2 The Consolidated Appropriations Act of 2018, Public Law 115-141, 115th Congress (Second Session), from which the committee’s Statement of Task was written, specified an interest in the time period of “fiscal years 2010 to 2017” (see Chapter 1).
Chapter 1), the committee concluded that patients receiving both medications are at a significantly higher risk for adverse outcomes than patients on opioids alone and that there was very limited evidence regarding opioid prescribing strategies specifically for patients receiving benzodiazepines. Thus, although the effects of opioid initiation and tapering on patient outcomes are important areas of inquiry, the focus on patients on benzodiazepines is more responsive to the Statement of Task and also addresses a particularly important sub-group of patients.
Finally, the Statement of Task specifically focuses on the outcomes of “deaths and suicides.” The committee believed that a focus on all-cause mortality appropriately reflects the fundamental importance to clinical decisions of evidence of increased or decreased risk of death from any cause. Additionally, the inclusion of suicide as an outcome separate from all-cause mortality is relevant to the concerns that patients, clinicians, and other stakeholders have about suicide among veterans generally as well as, specifically, among veterans with pain. The committee also discussed a number of other adverse outcomes as well as potential benefits of treatment that could be considered as secondary outcomes, depending on availability of data sources. Pain level, functional status, and quality of life are particularly important outcomes to assess in combination with the potential harms of treatment. However, the committee did not consider those outcomes as strictly relevant to the Statement of Task.
The committee chose to employ a “target trial” methodology; that is, it created a hypothetical randomized trial and described how it can be emulated (i.e., closely approximated) by an observational study to address the research question. A randomized trial is the preferred method for addressing causal questions about the comparative effectiveness and safety of medical treatments. For every such question, one can imagine a randomized trial that, if large enough and completed successfully, might answer that question. That hypothetical trial is referred to as the target trial. In practice, the target trial might be costly, infeasible, unethical, or simply too time consuming and thus not be practical to carry out. Thus, researchers must rely on available observational data (Hernán and Robins, 2016).
Researchers often perform observational analyses of health care databases when the target trial is not a viable option for answering the causal question of interest (Strom et al., 2012). Causal inference from observational databases can be viewed as an attempt to emulate the target trial. If the emulation is successful, then each analysis of the observational data is expected to yield the same effect estimates as the corresponding analysis
specified in the target trial, had it been successfully conducted. To guide decisions about which of several competing analytic strategies to use, causal analyses of observational data need to be evaluated with respect to how well they emulate the analyses that would occur within the corresponding target trial. Besides providing a structured process for the evaluation and criticism of observational studies, the target trial framework helps avoid common methodologic pitfalls of causal inference from observational data (Sterne et al., 2016).
An important thing to note is that observational data can only be used to emulate pragmatic target trials, that is, trials that compare treatment strategies currently in use and under the usual conditions in which they are applied in the real world (e.g., no placebo control, no blinding, no intensive monitoring). The necessarily pragmatic nature of the emulated target trial is not a limitation when the goal is comparing the effects of realistic treatment strategies in individuals who participate in decisions about their own health care.
This chapter outlines sample protocols for target trials that would quantify the effects of the initiation and discontinuation of opioids in the presence of benzodiazepines on all-cause mortality and suicide and that could be emulated using observational data collected by the VA. Such target trials could, in theory, be conducted, though it is debatable whether the standard of clinical equipoise3 is met for the research questions of interest. Additionally, recent trials indicate that many patients are unwilling to have their access to opioid medications subject to randomization. For example, 41 percent of VA opioid-naïve patients with a chronic pain diagnosis declined to participate in a randomized trial comparing stepped opioid and opioid-sparing protocols (Krebs et al., 2018). Not only does this mean that such randomized controlled trials would be difficult, if not impossible, to conduct and would take many years to provide answers, but also the degree of self-selection might limit their generalizability to the broader population of VA patients with chronic pain. Furthermore, such studies would need to have very large samples because the outcomes of interest (death and suicide) are rare. Therefore, observational studies have a particularly critical role—and, indeed, even possible advantages—in terms of generalizability.
The proposed research strategy has two basic steps: (1) asking the causal question, and (2) answering the causal question. Step 1 is aided by specifying the protocol of the target trial, and Step 2 is done by either conducting the target trial when possible or by emulating as closely as possible the target trial using observational data. Note that the data requirements
3Clinical equipoise, a concept taken from medical ethics, asserts that there should exist no decisive evidence that one of the treatment assignment groups is more effective or more safe than another (Cook and Sheets, 2011; London, 2017).
and data analysis procedures for Step 2 follow naturally from the explicit causal questions articulated in Step 1. The committee begins by describing Step 1.
The protocol of the target trial, like that of any other randomized trial, includes seven components: eligibility criteria, treatment strategies, treatment assignment, start and end of follow-up, outcomes, causal contrasts, and the statistical analysis plan (Hernán and Robins, 2016). Definitions of the baseline (or the start of the follow-up) and time-varying covariates, potential confounders, and the variables defining key sub-groups need to be included, as appropriate, within the seven components of the protocol. Therefore, the specification of the target trial needs to include the complete specification of each of these components (see Table 2-1).
TABLE 2-1 Key Components of the Target Trial Protocol
|Eligibility criteria||How the patient population is recruited into the trial.||All inclusion and exclusion criteria are based on characteristics ascertained exclusively at baseline.|
|Treatment strategies||Each of the clinical interventions that are to be compared.||The description needs to include the initial treatment as well as protocol-approved reasons for discontinuation or switching.|
|Treatment assignment||How participants will be assigned to each treatment strategy at baseline.||The assignment is randomized, possibly conditional on baseline prognostic factors. Patients will be aware of the treatment strategy to which they were assigned.|
|Start and end of follow-up||Define when the follow-up period starts and ends for each participant.||For each eligible individual, follow-up starts at baseline (the time of treatment assignment) and ends at death, outcome, loss to follow-up, or administrative end of follow-up.|
|Outcomes||Outcomes of interest and how to ascertain them.||If possible, include negative controls, i.e., outcomes that are known to be unaffected by the studied treatments.|
|Causal contrast||What comparative effects of the treatment strategies will be estimated.||The intention-to-treat effect (the comparative effect of being assigned to the treatment strategies at baseline) or per-protocol effect (the comparative effect of receiving the treatment as specified in the protocol).|
|Statistical analysis||How to estimate the intention-to-treat effect or per-protocol effect via intention-to-treat and per-protocol analyses that appropriately adjust for pre- and post-baseline prognostic factors associated with adherence and loss to follow-up.||Investigators should specify and measure the covariates potentially related to treatment choice, adherence, and outcomes at baseline and during the follow-up. Other variables that may need to be specified include those that define key sub-groups.|
SOURCE: Hernán and Robins, 2016.
Because there are many clinical questions about the effects of opioids and benzodiazepines that remain unanswered, multiple target trials might be proposed. As described above, this document focuses on two target trials, with each one comparing multiple clinical strategies in different patient populations. Specifically, one target trial examines the initiation of treatment for patients with chronic pain already taking benzodiazepines, and the other one examines strategies for the tapering of opioids in patients with chronic pain who are currently being treated with both benzodiazepines and opioids. Researchers may want to propose alternative target trials or variations of the committee’s proposed trials, for example, trials with different eligibility criteria or treatment strategies. Therefore, this document should not be viewed as a rigid description of the target trials that must be emulated before all other options, but rather as guidance on how to structure the specification of a target trial that precisely characterizes the research question. However, the committee believes the two example target trials are the minimum needed to address the key questions pertinent to the Statement of Task. Chapter 3 will focus on the adequacy of the available observational data to emulate the target trial and the data analyses required to carry out the emulation.
It is rare that one is able to emulate the ideal trial that would be of greatest interest. Rather, the available observational data will usually impose a number of constraints concerning the eligibility criteria, treatment strategies, available sample size (especially in sub-groups), and other components of the target trial protocol. Therefore, the specification of the design of the target trial and the associated analyses will typically be an iterative process in which investigators will learn which particular target trials may be reasonably emulated by the available observational data.
ADDRESSING THE RESEARCH QUESTION VIA A TARGET TRIAL THAT CAN BE EMULATED USING AVAILABLE OBSERVATIONAL DATA
The research question, “What were the effects of opioid initiation and tapering strategies, in the presence of benzodiazepines, in veterans, on all-cause mortality and suicide mortality from 2010 to 2017?” results in two distinct populations for study, namely: (1) veterans who enter the study on benzodiazepines but are not yet receiving opioids and suffering from pain-related conditions for which opioids were, during the study period, considered reasonable treatment options; and (2) veterans who enter the study actively treated with both benzodiazepines and opioids and for whom a reduction in their dosage of opioids was a treatment option. Thus, the committee proposes two trials, a treatment initiation trial and a tapering trial, with the first intended to identify optimal treatment initiation strategies for patients with pain who are taking benzodiazepines but are not actively being treated with opioids, and with the second intended to identify optimal approaches for reducing opioid dosages for patients already being treated with both opioids and benzodiazepines. It is important to note that the term “treatment” is used here to include a wide range of pharmacologic and non-pharmacological options.
The committee acknowledges that benzodiazepine initiation and tapering are also important and clinically relevant topics. However, given that the Centers for Disease Control and Prevention (CDC) guidelines recommend that “it might be safer and more practical to taper opioids first” (Dowell et al., 2016, p. 15), the committee chose to focus its proposed trials on the initiation and tapering of opioids and note that the committee’s model can be used as a template for other target trials and emulations, such as the initiation and tapering of benzodiazepines in the presence of opioids.
The committee applied the target trial protocol framework (see Table 2-1) to guide the design of the observational analyses proposed in Chapter 3. Table 2-2 describes considerations and options for the design components for each of the two target trials. The committee thought that a trial with the sample size necessary to study suicide and all-cause mortality (or the corresponding observational analyses) would be very likely to rely on administrative data sources rather than on intensive patient assessments. Thus, in designing the protocol for the target trials, the committee thought about the feasibility of emulating those trials using existing data. Nonetheless, Table 2-2 reflects a range of considerations for researchers to consult for finalizing the target trials and corresponding observational study designs that are not limited to options that could be based on available VA data. It is likely that limited availability and quality of observational data will result in a modification of the initial target trial specifications.
TABLE 2-2 Protocol Considerations for Treatment Initiation and Tapering Target Trials
|Treatment Initiation Trial||Tapering Trial|
Pain treatment modalities to consider:
Possible opioid dosage strategies to consider:
Randomization level can be:
(Same for initiation and tapering trials)
|Start and end of follow-up||
Considerations for censoring:
|Treatment Initiation Trial||Tapering Trial|
Function and quality of life
Health care utilization
(Same for initiation and tapering trials)
|Causal contrast||Intention-to-treat effect
(Same for initiation and tapering trials)
Population sub-groups that could be examined for potential effect modification:
The committee proposes first to characterize prescribing patterns along with the use of non-pharmacological treatment strategies and to use this information to guide the specification of the particular treatment strategies that would be compared in the target trials. As will be described in Chapter 3, the observational data can then be used to emulate a target trial that compares these existing treatment patterns in order to determine the relative mortality risk and whether risk varies across sub-groups defined by baseline characteristics.
While Table 2-2 provides the general considerations and a variety of options for each of the components of the target trials, a greater level of specificity in definition and in the selection of options is required to inform the translation of each target trial component into a trial emulation or observational data analysis strategy. In Table 2-3 the committee illustrates a suggested set of initial choices for the required specification of the target trials. The choices listed in this table should be considered preliminary because the specification of the target trial components is an iterative process, with insights from pilot analyses of the available observational data resulting in changes to the definitions and choices that can be incorporated into a feasible and valid analysis plan. Figure 2-1 illustrates the two trials described in Table 2-3.
TABLE 2-3 Proposed Specifications for Initiation and Tapering Target Trials
|Protocol Component||Initiation Target Trial||Tapering Target Trial|
|Eligibility criteria||Chronic pain diagnosisa
No prescriptions for opioids or non-aspirin NSAIDS in the past 90 days
Long-term benzodiazepine therapy (defined based on pilot data)
Individuals with serious illnessb
Individuals prescribed opioids used for treatment of opioid use disorder
Individuals with surgery or acute painful injury within the past 90 daysc
|Long-term opioid therapy defined as 3+ opioid fills ≥21 days apart in a ≥84-day period for ≥84-day supply (Larochelle et al., 2016)
Average opioid MMEd/day is ≥30 over the prior 84 dayse
Long-term benzodiazepine therapy (defined based on pilot data)
Individuals with serious illness
Individuals prescribed opioids for the treatment of opioid use disorder
Individuals with surgery or acute painful injury within the 90 days prior to baseline
|Protocol Component||Initiation Target Trial||Tapering Target Trial|
Participants who cannot tolerate their assigned dosage change will be excused from following their assigned strategy. Percentage of taper is relative to opioid dose at baseline and is calculated over the next 3 months. After that period, dosage is left to the physician’s discretion.
|Treatment assignment||Individual randomization, stratified on baseline dose|
|Start and end of follow-up||Start of follow-up (baseline): start of treatment for chronic pain, defined as being dispensed one or more prescriptions of an opioid or NSAID for at least a 30-day supply over a 30-day period (this could be across multiple prescriptions), and also having a chronic pain diagnosis.
End of follow-up: the earliest of 18 months,g death, or administrative end of follow-up (end of the study).
|Start of follow-up (baseline): time of assignment to a treatment strategy.
End of follow-up: the earliest of 6 months,h death, or administrative end-of-follow-up (end of the study).
|Protocol Component||Initiation Target Trial||Tapering Target Trial|
|Statistical analysis||Intention-to-treat analysis: check for balance on key variables, e.g., mental health diagnoses and substance use disorders.
Per-protocol analysis: patients will be censored at the time they deviate from their assigned strategy. To adjust for the potential selection bias induced by censoring, inverse probability weighting will be used. The weights will be a function of the baseline and post-baseline (time-varying) confounders.
Both analyses may require further adjustment for selection bias due to loss to follow-up.
Pre-specified sub-groups to be examined for potential effect modification include, e.g., pain severity, history of overdose, history of suicide attempt, non-suicide death (for the suicide death analysis).
a The intention of this definition is to exclude opioids and NSAIDs prescribed for acute pain. However, researchers should consider that there might be a large proportion of veterans prescribed opioids for whom there is not a chronic pain diagnosis (Edelman et al., 2013).
b Serious illness is defined by Kelley and Bollens-Lund (2018) as a health condition that carries a high risk of mortality and negatively affects a person’s daily functioning. The committee recommends operationalizing this as any of the following conditions: cancer, chronic obstructive pulmonary disease, congestive heart failure, dementia, or severe neurologic disorder (e.g., amyotrophic lateral sclerosis, multiple sclerosis).
c 90 days was chosen to minimize likelihood of opioids being prescribed for acute rather than chronic pain conditions. However, the committee acknowledges that the choice of 90 as opposed to 30 or 60 is arbitrary.
d MME = morphine milligram equivalent.
e This threshold was used because labeling for OxyContin extended release defines “opioid tolerant” as consuming 30 MME/day. Researchers might consider a lower dose threshold if the purpose is to include anyone who could be considered for a taper.
f Speed of tapering: there is a lack of primary literature on the optimal rate of tapering speed (i.e., rate of dosage decrease per week/month). Within the context of concomitant opioid and benzodiazepine use and likely psychiatric comorbidity, a more conservative approach would be prudent.
g The committee felt that the 18-month timeline balanced the desire for a longer length of follow-up than prior initiation studies with the fact that there would be a greater degree of non-adherence from the assigned treatment group for longer lengths of follow-up.
h The committee felt that the 6-month timeline balanced a desire for a longer length of follow-up with potential for non-adherence and a concern that suicide as a relatively short-term outcome in tapering studies.
Eligibility criteria determine which patients can be enrolled into a trial. The eligibility criteria for each of the example target trials are intended to define a patient population for which a comparison of the outcomes associated with alternative management strategies would be informative with respect to the research question. Additionally, the treatment strategies to be compared must be realistic options for patients meeting the eligibility criteria. For example, for a comparison of outcomes under different approaches for managing patients with long-term opioid and benzodiazepine use, the eligibility criteria should define a population with such long-term use who would reasonably or historically have been eligible for treatment with different maintenance or tapering strategies (e.g., excluding patients requiring palliative care or hospice). To define a population in a manner as similar as possible to that used in a true randomized trial, all eligibility criteria must be defined at “time zero” of follow-up, that is, at the time of treatment assignment.
While enrollment criteria used in traditional explanatory trials—as opposed to pragmatic trials—are generally intended to define a relatively homogeneous patient population that is likely to demonstrate little variability in response to treatment, pragmatic trials tend to use fewer restrictive enrollment criteria (Sox and Lewis, 2016). Such a population is expected to have clinically distinct sub-groups that may differ in their underlying prognoses or event rates and in their responses to alternative treatment strategies. Here, with the use of a pragmatic strategy the enrollment criteria are purposely broad, with the intent of identifying a wide range of patients who may have been, or may be in the future, treated with opioids, benzodiazepines, or both. This strategy also maximizes the number of patients to whom the results will be applicable.
For the purposes of assigning patients to clinically relevant sub-groups for statistical risk adjustment or for the implementation of planned statistical analyses intended to emulate those defined by the target trial protocol, it will be necessary to measure a wide variety of characteristics or covariates, both at baseline and during the period of follow-up. These characteristics or covariates may be particular to a given patient, to the practitioner or the VA practice setting, to geography, or to other factors that potentially affect treatment selection and outcomes.
Treatment Initiation Target Trial
For the treatment initiation trial, the committee defined the eligible patient population to be patients with a chronic pain diagnosis currently
being prescribed benzodiazepines but not opioids. The goal was to define a population for which different patterns in the initiation of opioid treatment—including no initiation—in the presence of long-term benzodiazepine use would be relevant treatment options during 2010–2017. Specifically, eligible patients must have had a chronic pain diagnosis and have had a period of 90 or more days since they had last taken opioids, based on pharmacy fill data. Furthermore, the patients must have demonstrated the long-term and stable use of benzodiazepines during that same 90-day time period. Long-term and stable use was required because the question most relevant to the Statement of Task regarded opioid use that is concomitant with benzodiazepine use, and concurrent changes in benzodiazepine use would obscure the effects of opioids on adverse outcomes. The definition of long-term and stable use of benzodiazepines is somewhat inconsistent in existing literature and therefore would need to be determined based on reasonable cut-points from pilot VA data.
Tapering Target Trial
For the treatment tapering trial, the committee defined the eligible population as those VA patients prescribed both opioids and benzodiazepines on a long-term basis, excluding patients in whom attempts at opioid tapering are unlikely or contraindicated, such as patients with terminal medical conditions requiring palliative care or hospice. It should be noted that the taper would be voluntary, and frequent patient monitoring for psychiatric comorbidities during the taper would be critical.
Opioids and benzodiazepines have distinct indications for use. Long-term opioid therapy is primarily used to treat chronic pain.4 It is important to note that long-term opioid therapy always begins as an initial prescription, likely intended to cover 1 month or less (rather than an initial prescription for long-term therapy). That initial prescription may be to treat acute pain that eventually becomes chronic or to treat pain that is already chronic. In either case, opioids may be continued long-term.
Benzodiazepines carry indications for panic disorder, generalized anxiety disorder, insomnia, detoxification from alcohol, and general complaints of anxiety accompanying other psychiatric conditions such as depression, posttraumatic stress disorder (PTSD), and adjustment disorders (Ciraulo and Nace, 2000; Katzman et al., 2014; Nichols et al., 2019). The long-term
4 Buprenorphine and methadone are predominantly used to treat opioid use disorder.
use of benzodiazepines is currently discouraged in favor of using antidepressant medications, behavioral treatments, or psychotherapy (Driot et al., 2019; Platt et al., 2016) for many of these indications, but, nevertheless, benzodiazepines are still commonly prescribed for long periods of time by many clinicians (Tanguay et al., 2018).
Preliminary descriptive analyses of available data will be required to determine and finalize the selection of the treatment strategies for each target trial in order to ensure that the analyses intended to emulate the target trials include sufficient numbers of patients who received each strategy across the VA during the 2010–2017 period. After those preliminary descriptive analyses, strategies may include any combination of specific patterns within these broad categories: (1) opioid and benzodiazepine pharmacotherapy (together or separately), (2) non-opioid, non-benzodiazepine pharmacotherapies, (3) non-pharmacological therapies, and (4) changes in treatment monitoring and frequency.
As patients will repeatedly present to care, those treatment strategies can be considered as options at every clinic visit. For example, in usual care, opioids might be continued at one visit, physical therapy started at the next visit, and opioids reduced at the next visit, and so on. All of the treatment strategies may be used independently of each other, although some combinations will be more or less common in actual practice. The frequency of visits will also vary depending on the types of strategies pursued and each patient’s condition. In the next sections, the committee defines the general treatment strategies to be considered in the treatment initiation and tapering target trials.
Treatment Initiation Target Trial
The committee considered major categories of pharmacologic therapies that are used for pain, which would include anticonvulsants, muscle relaxants, antidepressants, medical cannabis, and topical therapies. Patients could receive one or more of those treatments at baseline, and preliminary analyses could determine whether patients receiving those medications should be included or excluded from the observational study. The committee also considered major categories of non-pharmacological strategies used to address pain, including behavioral interventions, complementary and alternative therapies, exercise therapy, yoga, and physical therapies.
A clinician could decide to increase or decrease the frequency of visits, monitoring, and other treatments based on the patient’s response to treatment. The frequency of treatment visits and types of monitoring activities could have a significant impact on the treatment, regardless of the specific treatment strategy, and could correlate with treatment outcomes. The
approaches to monitoring could include changing the frequency of visits or using such monitoring strategies as urine drug testing or pill counts; these approaches might vary by site or clinic. The ability to include those considerations in the definition of the target trial will be determined by the availability of suitable data in existing VA datasets.
To study the effects of initiating an opioid for chronic pain on mortality outcomes, various approaches could be possible; however, the committee believes that the most useful study would compare opioids to nonsteroidal anti-inflammatory drugs (NSAIDs).5 A comparison group receiving an active treatment will be more similar to the group treated with opioids in characteristics such as pain severity and access to pain management services, in contrast to a group with a chronic pain diagnosis receiving no treatment. The committee chose non-aspirin NSAIDs (henceforth “NSAIDs”) as the comparison treatment, although other comparisons might also be appropriate. Opioids and NSAIDs are both used routinely in the treatment of chronic pain, whereas many other non-opioid analgesics have multiple indications (e.g., gabapentanoids). Additionally, by comparing medications to one another, the study has the advantage of using parallel measurement approaches for both treatment strategies.
In the more specific clinical context sought to be addressed by this study, namely a patient with a chronic pain diagnosis not having used an opioid during the past 90 days while receiving chronic treatment with a benzodiazepine, a clinician could begin long-term therapy with an opioid, an NSAID, or both. The committee defined beginning opioid treatment (or NSAID treatment) for chronic pain as being dispensed one or more prescriptions of an opioid (or NSAID) for at least a 30-day supply over a 30-day period and having a chronic pain diagnosis. Other definitions may be considered. Furthermore, there is a wide range of other pharmacologic and non-pharmacological pain treatment strategies that could be used alone or in combination.
Tapering Target Trial
The goal of tapering is to safely reduce an opioid dosage. Opioid treatment guidelines recommend that the prescriber conduct frequent reassessments of the benefits and potential harms of opioid therapy (Dowell et al., 2016). If the harms have the potential to outweigh the benefits for a particular patient, the prescriber should consider whether it is appropriate to reduce the opioid dosage. The goal of the dosage reduction can be to achieve a complete discontinuation of opioid use after some period
5 All references to NSAIDs refer to non-aspirin NSAIDs.
(sometimes termed an “opioid taper”), but maintaining treatment at a lower daily dosage might also be the goal. The decision to taper might be due to a variety of factors, including concerning patient behaviors (e.g., overtaking the medication, illicit substance use) or the provider’s desire to reduce opioid-related risk. In some cases a prescriber might believe that an abrupt discontinuation is clinically indicated for a particular patient, but there is no consensus among opioid prescribers on when abrupt discontinuation is appropriate, and there is concern that such abrupt discontinuation may increase the potential for harm in the context of physical or psychological dependence (Dowell et al., 2016). Given the evidence of harm associated with concurrent opioid and benzodiazepine use, including risk of fatal respiratory depression (Dowell et al., 2016), many treatment guidelines stress the need for frequent reassessment, with a goal of eventually discontinuing either the opioid or the benzodiazepine (Dowell et al., 2016; VA/DoD, 2017). As previously noted, the committee focused on the tapering of opioids, because CDC recommends that as the safer and more practical first step (Dowell et al., 2016).
Opioid Dosage Measurement
In the tapering target trial, patients would be eligible for inclusion into the study after their prescribed daily opioid dosage had reached a level that would be likely to induce opioid dependence. This is because opioid use at a lower dosage would be unlikely to require a slow dosage reduction in an effort to avoid withdrawal symptoms. Each patient’s care team would be responsible for implementing the treatment strategy to which the patient was randomized over a 3-month period as well as any decisions to deviate from that strategy during the follow-up period. Patients may be prescribed opioids by other clinicians (e.g., surgeons, those in other health care systems) in ways inconsistent with the discontinuation or tapering strategies, but not the continuation strategy, representing an additional form of non-adherence that is often measurable from claims records. The following dosage strategies should be considered:
- No dosage reduction: Continue opioid dosage at the same level (or ≤5 percent reduction)6; increase as indicated for symptoms or tolerance; taper/discontinue if not tolerated. Continuation may be measured as prescribed daily dosage that is the same or greater for month-to-month change throughout the 3 months.
6 Reductions of less than 5 percent are considered non-meaningful variation based on measurement.
- Slow dosage reduction: Reduce dosage by 5 to 10 percent per month on average over 3 months;7 stop taper and resume or increase dosage if indicated by pain level or if the risks of discontinuation outweigh benefits. This strategy may be identified by a reduction of, on average, 5 to 10 percent each month during the 3-month window and by the absence of month-to-month decreases in any given month consistent with a moderate to fast dosage reduction or complete discontinuation, described below.
- Moderate to fast dosage reduction: Continue treatment but reduce dosage by more than 10 percent in 1 month; stop taper and resume or increase dosage if indicated by pain level or if the risks of discontinuation outweigh benefits (Darnall et al., 2018; Dowell et al., 2016). This strategy may be identified by a reduction of greater than 10 percent (but less than 100 percent, which would be a complete discontinuation, described below) from 1 month to the next at least once in a 3-month window.
- Abrupt discontinuation: Stop taking opioids completely; resume use if indicated by pain level or risks of discontinuation outweigh benefits. As an example, this pattern may be measured as a lapse in days covered by prescriptions of at least 14 days within a 3-month window.
Individuals in the target trial would be randomly assigned to one of the treatment strategies, stratified by baseline dose. As in most pragmatic trials, the assignment would be non-blinded; that is, both patients and their treating physicians would be aware of the assigned treatment strategy. In the emulation of these target trials, treatment assignment is observed based on treatment records, requiring definitions of treatment groups that allow the creation of meaningful treatment groups and also allow for non-adherence to the assigned treatment that would occur in the target trial.
7Darnall et al. (2018) uses 5 percent “for up to two dose reductions in one month” as an initial tapering speed, then no more than 10 percent per week. The CDC guideline (Dowell et al., 2016) also notes that tapers slower than 10 percent per week are likely better tolerated than faster tapers and that tapers may have to be started and stopped. Additionally, it is not possible to assess tapering per week when prescriptions are often written for 30 days. Thus, the committee suggests a 5 percent cut-point for slow dosage reductions and 10 percent for fast dosage reductions.
For the participants in the target trial, follow-up would start at the time of treatment assignment (for the initiation study, defined as being dispensed one or more prescriptions of an opioid or NSAID for at least a 30-day supply over a 30-day period; for the tapering study, defined as starting one of the defined tapering strategies) and would end at 18 months for the initiation trial or at 6 months for the tapering trial, or when the patient dies, or when the study ends.
For the proposed research to be of value to clinicians and patients, it will be necessary for it to include outcomes that are relevant to treatment decisions. As mentioned previously, the committee chose all-cause mortality and death from suicide as the primary outcomes of interest for the proposed target trials. However, in the case of the opioid initiation and tapering studies, many other secondary outcomes could be measured that are related to the potential benefits of treatment as well as to the potential harms.
Other Secondary Measures of Potential Benefits of Treatment
There are existing and common measures, often used in prospective clinical trials, of many outcomes relevant to the use of opioids and benzodiazepines. These outcomes include pain, anxiety, symptoms of PTSD, and the common functions of daily living.
- Pain relief: Patients typically self-report pain intensity level on an 11-point scale (the numeric rating scale ranging from 0 to 10) in the course of the clinical management of pain. A 30 percent change in pain intensity level is considered a clinically significant change (Hanley et al., 2006).
- Anxiety reduction: Several of the major indications for benzodiazepines are forms of anxiety disorders. Anxiety level is often measured with the Generalized Anxiety Disorder (GAD-7) scale. The GAD-7 represents an anxiety measure based on seven items scored from 0 to 3 based on self-report (Jordan et al., 2017).
- PTSD symptoms: Although PTSD is not considered an indication for benzodiazepines under VA clinical guidelines, is it not uncommon for veterans with PTSD to receive benzodiazepines. The PTSD symptom level is routinely measured in VA primary care with the PTSD Checklist. Additionally, some patients receive
benzodiazepines without a relevant diagnosis in their medical records.
- Improved functioning: For many patients, the ability to carry out the daily activities of living is the goal of pain or anxiety management. General scales of function, such as the Short Form 12 (SF-12), provide an option to measure this domain across treatments and indications. The SF-12 consists of 12 questions that measure eight health domains in order to assess physical and mental health (Huo et al., 2018). For pain specifically, the three-item PEG scale developed by Krebs and colleagues (2009) incorporates measures of both pain intensity and pain-related functioning and is routinely used in the clinical management of pain in the VA. Items in the PEG scale assess average pain intensity (P), interference with enjoyment of life (E), and interference with general activity (G).
- Health care use: Effective pain management would likely reduce the need for increased levels of medical care, while less effective management (either under-treatment or over medication) would likely result in greater use of health care.
Potential Harms of Treatment
The committee also considered the potential adverse effects of treatment, such as measures of mortality, suicide, and overdose.
- All-cause mortality: A fundamental concern about any treatment is the risk of premature mortality. Assessing all-cause mortality as an outcome also has the advantage of avoiding problems with misclassification and missing data inherent to cause-specific mortality outcomes. Cause-of-death codes can be used to differentiate definite self-harm from possible self-harm in order to avoid missing potential cases of suicide.
- Suicide: Suicide is associated with pain as well as with higher prescribed dosages of opioids. Clinicians concerned about excessive restrictions on access to prescribed opioids have hypothesized that patients who have been on long-term opioid therapy are at an increased risk for suicide during and after opioid discontinuation (Darnall et al., 2018; HP3, 2019). In April 2019 the Food and Drug Administration released a drug safety announcement to communicate reports of suicide in patients physically dependent on opioid pain medicines suddenly having their opioids discontinued or the dosages rapidly decreased. The announcement posits that rapid discontinuation can result in uncontrolled pain
or withdrawal symptoms, which in turn can lead patients to seek other sources of opioid pain medicines, including illicit opioids.
- Overdose: Unintentional opioid overdose is the potential risk of treatment that has driven much of the effort to reduce potentially excessive opioid prescribing, although it is a relatively rare outcome. Evidence that higher prescribed daily dosages are associated with a greater risk of opioid overdose may have led some physicians to taper patients in an effort to reduce overdose risk. Some advocates for a measured approach to opioid prescribing reductions have hypothesized that the population-level increases in heroin and illegally manufactured fentanyl overdoses are a result of patients transitioning to illegal opioid use after opioid discontinuation (Alpert et al., 2018; Cicero and Ellis, 2015; Cicero et al., 2012; Evans et al., 2019; Larochelle et al., 2015).
Other potential harms of treatments include increased depressive symptoms or suicidal ideation precipitated by non-consensual opioid reduction, incident psychiatric illness, opioid use disorder/opioid dependence, and multi-substance use disorder (SUD) (Glanz et al., 2019). Researchers should also consider the potential benefits and harms of NSAIDs in an effort to understand the trade-offs between the different classes of drugs.
For each of the outcomes listed above, there are two causal contrasts of interest: the intention-to-treat effect and the per-protocol effect.
The intention-to-treat effect is the effect of being assigned to the treatment strategies at baseline, regardless of the treatment that is actually received. Because most individuals will initiate the strategies at the time of randomization, the intention-to-treat effect is effectively the effect of the initiation of the treatment strategy. The magnitude of the intention-to-treat effect depends on the patterns of non-adherence, the association between adherence and prognosis, and other deviations from protocol that occur during the trial. Two trials of the same treatment strategies conducted in the same population could have different intention-to-treat effects if their adherence patterns differed, and both would be internally valid effects of the assignment to treatment.
The per-protocol effect is the effect of receiving the treatment strategies throughout the follow-up as specified in the study protocol (e.g., without non-adherence). The numerical value of the per-protocol effect does not depend on the particular patterns of deviation from protocol that occur during the trial. Two trials of the same treatment strategies conducted in the same population could have different intention-to-treat effects if the
adherence patterns differed, but they should have the same per-protocol effect.
The intention-to-treat effect estimates capture the impact of initiating one treatment strategy versus another and, thus, is ideal for informing recommendations for initial treatment selection. That is true because intention-to-treat estimates are agnostic to post-randomization treatment decisions—including discontinuation of the treatment strategies of interest, use of concomitant therapies, or any other deviations from protocol (Hernán and Hernández-Díaz, 2012). The intention-to-treat effect combines the effect of the treatment under study with that of other behavioral changes in the patient or physician triggered by the assignment itself. The intention-to-treat effect is widely used because intention-to-treat analyses preserve the randomized assignment in cases of non-adherence and thus protect against confounding (Ranganathan et al., 2016).
On the other hand, the intention-to-treat estimate may be hard to interpret for patients and clinicians who may desire an estimate of the “pure” treatment effect, that is, the effect that would hypothetically be associated with perfect compliance with the assigned treatment strategy. Importantly, when evaluating the harms of treatment, the intention-to-treat effect incorporates both the harms of the treatment if taken as intended and the willingness or ability of patients to adhere to the prescribed therapy. If the goal is to understand the risk of harm for a patient who is adherent, then the intention-to-treat estimate is an inappropriate choice because a risky treatment may appear less risky if patients are poorly adherent. The per-protocol is generally the more conservative estimate when estimating harm (perhaps giving an estimate of harm that is higher than seen in the population as a whole), while the intention-to-treat estimate is generally the conservative estimate when evaluating benefits (perhaps giving an estimate of benefit that is lower than seen in patients who are highly adherent) (Hernán and Hernández-Díaz, 2012; Sheiner and Rubin, 1995). When patient adherence is excellent, there will be little difference between the two estimates. In this setting of evaluating the harms of treatment, the per-protocol estimate directly addresses the question of interest and should be used. There is empirical evidence that, in some settings, the per-protocol effect is closer than the intention-to-treat effect to the sort of effect that patients and investigators are mostly interested in discovering from pragmatic trials, namely, which treatment would be more effective if taken as indicated in the protocol (Hernán and Robins, 2017; Murray et al., 2018). The per-protocol effect is more relevant for a patient who intends to be adherent; an estimate of the “pure” effect of the treatment is easier for a patient and clinician to discuss and understand than an estimate that intermingles the “pure” effect and adherence (Hernán and Robins, 2017). When the intention-to-treat effect is said to be “biased toward the null” or conservative, the implication is that
the intention-to-treat effect is a biased estimate of the per-protocol effect (Murray et al., 2018). An added advantage of the per-protocol effect is that its interpretation does not depend on a trial-specific degree of adherence, which makes it a more transportable effect. However, the potential benefit that can be received by patients on average will be best represented by the intention-to-treat estimate rather than the per-protocol effect.
Two separate sets of statistical analyses would be conducted after the completion of the target trial, one to estimate the intention-to-treat effect and another one to estimate the per-protocol effect.
An analysis aimed at estimating the intention-to-treat effect is referred to as an intention-to-treat analysis. In a large pragmatic trial with complete follow-up, the intention-to-treat analysis is straightforward: compare the observed outcome distributions between trial arms. That is, under those conditions, the intention-to-treat effect can be validly estimated without an adjustment for prognostic factors. In contrast, as discussed in the next chapter, observational analyses used to emulate the target trial that attempt to emulate intention-to-treat analyses will generally need to be adjusted for prognostic factors that confound the effect of treatment on the outcome.
An analysis aimed at estimating the per-protocol effect is referred to as a per-protocol analysis. Unlike intention-to-treat analyses, per-protocol analyses generally require adjustment for pre- and post-randomization prognostic factors that predict adherence to the protocol. That is, per-protocol analyses of randomized trials can be viewed as observational analyses, which require the same methods and rely on the same assumptions as the analyses of observational datasets.
The protocol of the target trial would therefore need to pre-specify the following three sets of adjustment variables: (1) pre-randomization prognostic factors that (if imbalanced) would need to be adjusted for in intention-to-treat analyses, (2) pre- and post-randomization prognostic factors to adjust for baseline and time-varying confounding in per-protocol analyses, and (3) pre- and post-randomization prognostic factors to adjust for potential selection bias due to loss to follow-up in both intention-to-treat and per-protocol analyses. Set 1 will generally be included in set 2. The variables in sets 2 and 3 will also be necessary to adjust for confounding and selection bias in observational analyses, which will be described in Chapter 3.
In addition, both intention-to-treat and per-protocol analyses could, in the target trial, be conducted in the following pre-specified sub-groups of patients:
- Patients with coexisting medical illnesses, especially those with treatments that limit the safety of non-opioid analgesics;
- Patients with psychiatric illness, particularly those with poorly controlled mood symptoms or a history of suicide attempts;
- Patients with SUD, particularly in those patients with concern for active substance use;
- Patients who prefer certain treatment strategies over others (e.g., patients who do not want to engage in behavioral approaches and who prefer medications); and
- Patients with a pain treatment history and a history of overdose with prescribed opioids, benzodiazepines, or other sedating medications (e.g., gabapentin) or illicit drugs (e.g., heroin).
It should be noted that the purpose of the sub-group analyses is to identify treatment effect heterogeneity across different groups of patients rather than to adjust for confounding and selection bias. Some of the variables used to explore this potential effect modification might also be included in the three sets of variables needed to adjust for confounding and selection bias. The sub-groups listed here will be explained in further detail in Chapter 3 in the discussion on why those same variables should be adjusted for confounding in the observational emulation of the target trials.
Alpert, A., D. Powell, and R. L. Pacula. 2018. Supply-side drug policy in the presence of substitutes: Evidence from the introduction of abuse-deterrent opioids. American Economic Journal: Economic Policy 10(4):1–35.
Cicero, T. J., and M. S. Ellis. 2015. Abuse-deterrent formulations and the prescription opioid abuse epidemic in the United States: Lessons learned from oxycontin. JAMA Psychiatry 72(5):424–429.
Cicero, T. J., M. S. Ellis, and H. L. Surratt. 2012. Effect of abuse-deterrent formulation of oxycontin. New England Journal of Medicine 367(2):187–189.
Ciraulo, D. A., and E. P. Nace. 2000. Benzodiazepine treatment of anxiety or insomnia in substance abuse patients. American Journal of Addiction 9(4):276–279; discussion 280–284.
Cook, C., and C. Sheets. 2011. Clinical equipoise and personal equipoise: Two necessary ingredients for reducing bias in manual therapy trials. Journal of Manual and Manipulative Therapy 19(1):55–57.
Darnall, B. D., M. S. Ziadni, R. L. Stieg, I. G. Mackey, M. C. Kao, and P. Flood. 2018. Patient-centered prescription opioid tapering in community outpatients with chronic pain. JAMA Internal Medicine 178(5):707–708.
Dowell, D., T. M. Haegerich, and R. Chou. 2016. CDC guideline for prescribing opioids for chronic pain—United States, 2016. JAMA 315(15):1624–1645.
Driot, D., S. Ouhayoun, F. Perinelli, C. Grezy-Chabardes, J. Birebent, M. Bismuth, and J. Dupouy. 2019. Non-drug and drug alternatives to benzodiazepines for insomnia in primary care: Study among GPS and pharmacies in a southwest region of France. Therapie. doi: 10.1016/j.therap.2019.03.004. [Epub ahead of print].
Edelman, E. J., K. Gordon, W. C. Becker, J. L. Goulet, M. Skanderson, J. R. Gaither, J. Brennan Braden, A. J. Gordon, R. D. Kerns, A. C. Justice, and D. A. Fiellin. 2013. Receipt of opioid analgesics by HIV-infected and uninfected patients. Journal of General and Internal Medicine 28(1):82–90.
Evans, W. N., E. M. J. Lieber, and P. Power. 2019. How the reformulation of Oxycontin ignited the heroin epidemic. Review of Economics and Statistics 101(1):1–15.
Frank, J. W., T. I. Lovejoy, W. C. Becker, B. J. Morasco, C. J. Koenig, L. Hoffecker, H. R. Dischinger, S. K. Dobscha, and E. E. Krebs. 2017. Patient outcomes in dose reduction or discontinuation of long-term opioid therapy: A systematic review. Annals of Internal Medicine 167(3):181–191.
Glanz, J. M., I. A. Binswanger, S. M. Shetterly, K. J. Narwaney, and S. Xu. 2019. Association between opioid dose variability and opioid overdose among adults prescribed long-term opioid therapy. JAMA Network Open 2(4):e192613.
Hanley, M. A., M. P. Jensen, D. M. Ehde, L. R. Robinson, D. D. Cardenas, J. A. Turner, and D. G. Smith. 2006. Clinically significant change in pain intensity ratings in persons with spinal cord injury or amputation. Clinical Journal of Pain 22(1):25–31.
Hernán, M. A., and S. Hernández-Díaz. 2012. Beyond the intention-to-treat in comparative effectiveness research. Clinical Trials 9(1):48–55.
Hernán, M. A., and J. M. Robins. 2016. Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology 183(8):758–764.
Hernán, M. A., and J. M. Robins. 2017. Per-protocol analyses of pragmatic trials. New England Journal of Medicine 377(14):1391–1398.
HP3 (Health Professionals for Patients in Pain). 2019. Health professionals call on the CDC to address misapplication of its guideline on opioids for chronic pain through public clarification and impact evaluation. https://healthprofessionalsforpatientsinpain.org/the-letter-1 (accessed May 12, 2019).
Huo, T., Y. Guo, E. Shenkman, and K. Muller. 2018. Assessing the reliability of the Short Form 12 (SF-12) health survey in adults with mental health conditions: A report from the Wellness Incentive and Navigation (WIN) study. Health and Quality of Life Outcomes 16(1):34.
Jordan, P., M. C. Shedden-Mora, and B. Löwe. 2017. Psychometric analysis of the Generalized Anxiety Disorder scale (GAD-7) in primary care using modern item response theory. PLOS ONE 12(8):e0182162.
Katzman, M. A., P. Bleau, P. Blier, P. Chokka, K. Kjernisted, M. Van Ameringen, Canadian Anxiety Guidelines Initiative Group on behalf of the Anxiety Disorders Association of Canada/Association Canadienne des troubles anxieux and McGill University, M. M. Antony, S. Bouchard, A. Brunet, M. Flament, S. Grigoriadis, S. Mendlowitz, K. O’Connor, K. Rabheru, P. M. A. Richter, M. Robichaud, and J. R. Walker. 2014. Canadian clinical practice guidelines for the management of anxiety, posttraumatic stress and obsessive-compulsive disorders. BMC Psychiatry 14(Suppl 1):S1.
Kelley, A. S., and E. Bollens-Lund. 2018. Identifying the population with serious illness: The “denominator” challenge. Journal of Palliative Medicine 21(S2):S7–S16.
Krebs, E. E., K. A. Lorenz, M. J. Bair, T. M. Damush, J. Wu, J. M. Sutherland, S. M. Asch, and K. Kroenke. 2009. Development and initial validation of the PEG, a three-item scale assessing pain intensity and interference. Journal of General Internal Medicine 24(6):733–738.
Krebs, E. E., A. Gravely, S. Nugent, A. C. Jensen, B. DeRonne, E. S. Goldsmith, K. Kroenke, M. J. Bair, and S. Noorbaloochi. 2018. Effect of opioid vs. nonopioid medications on pain-related function in patients with chronic back pain or hip or knee osteoarthritis pain: The space randomized clinical trial. JAMA 319(9):872–882.
Larochelle, M. R., F. Zhang, D. Ross-Degnan, and J. F. Wharam. 2015. Rates of opioid dispensing and overdose after introduction of abuse-deterrent extended-release oxycodone and withdrawal of propoxyphene. JAMA Internal Medicine 175(6):978–987.
Larochelle, M. R., J. M. Liebschutz, F. Zhang, D. Ross-Degnan, and J. F. Wharam. 2016. Opioid prescribing after nonfatal overdose and association with repeated overdose: A cohort study. Annals of Internal Medicine 164(1):1–9.
London, A. J. 2017. Equipoise in research: Integrating ethics and science in human research. JAMA 317(5):525–526.
Murray, E. J., E. C. Caniglia, S. A. Swanson, S. Hernández-Díaz, and M. A. Hernán. 2018. Patients and investigators prefer measures of absolute risk in subgroups for pragmatic randomized trials. Journal of Clinical Epidemiology 103:10–21.
Nichols, T. A., S. Robert, D. J. Taber, and J. Cluver. 2019. Alcohol withdrawal-related outcomes associated with gabapentin use in an inpatient psychiatric facility. Mental Health Clinician 9(1):1–5.
Platt, L., A. Irene Whitburn, A. G. Platt-Koch, and R. Koch. 2016. Nonpharmacological alternatives to benzodiazepine drugs for the treatment of anxiety in outpatient populations: A literature review. Journal of Psychosocial Nursing and Mental Health Services 54(8):35–42.
Ranganathan, P., C. S. Pramesh, and R. Aggarwal. 2016. Common pitfalls in statistical analysis: Intention-to-treat versus per-protocol analysis. Perspectives in Clinical Research 7(3):144–146.
Sheiner, L. B., and D. B. Rubin. 1995. Intention-to-treat analysis and the goals of clinical trials. Clinical Pharmacology & Therapeutics 57(1):6–15.
Sox, H. C., and R. J. Lewis. 2016. Pragmatic trials: Practical answers to “real world” questions. JAMA 316(11):1205–1206.
Sterne, J. A., M. A. Hernán, B. C. Reeves, J. Savovi , N. D. Berkman, M. Viswanathan, D. Henry, D. G. Altman, M. T. Ansari, I. Boutron, J. R. Carpenter, A. W. Chan, R. Churchill, J. J. Deeks, A. Hróbjartsson, J. Kirkham, P. Jüni, Y. K. Loke, T. D. Pigott, C. R. Ramsay, D. Regidor, H. R. Rothstein, L. Sandhu, P. L. Santaguida, H. J. Schünemann, B. Shea, I. Shrier, P. Tugwell, L. Turner, J. C. Valentine, H. Waddington, E. Waters, G. A. Wells, P. F. Whiting, and J. P. Higgins. 2016. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Online) 355:i4919.
Strom, B. L., S. E. Kimmel, and S. Hennessy. 2012. Pharmacoepidemiology. Hoboken, NJ: Wiley.
Tanguay Bernard, M. M., M. Luc, J. D. Carrier, L. Fournier, A. Duhoux, E. Cote, O. Lessard, C. Gibeault, C. Bocti, and P. Roberge. 2018. Patterns of benzodiazepines use in primary care adults with anxiety disorders. Heliyon 4(7):e00688.
VA/DoD (Department of Veterans Affairs and Department of Defense). 2017. VA/DOD clinical practice guideline for opioid therapy for chronic pain. https://www.healthquality.va.gov/guidelines/Pain/cot/VADoDOTCPG022717.pdf (accessed July 19, 2019).
This page intentionally left blank.