National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×

Institute for Laboratory Animal Research

Division on Earth and Life Studies

500 Fifth Street, NW

Washington, DC 20001

Phone: (202) 334-2347

December 22, 2020

Joan T. Richerson, M.S., D.V.M., DACLAM

Assistant Chief Veterinary Medical Officer

Tennessee Valley Healthcare System

U.S. Department of Veterans Affairs

Nashville, TN 37212

Dear Dr. Richerson,

This letter report describes the work of the Committee on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder.

The committee received the following documents on November 10, 2020, for review:

  • ORD Monograph 1 cover letter 11 10 20;
  • Monograph 1 (2nd draft) submitted to NASEM 11 10 20;1
  • Monograph 1 (2nd draft) tracked changes;
  • Monograph 1 VA Responses to NASEM Committee Report 11 10 20.

The committee thanks the study team at the Department of Veterans Affairs (VA) for providing the additional documents to assist in the committee’s review of the revised monograph and for its careful attention to the suggestions and concerns noted in the committee’s first report, Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. The committee found the “Monograph 1 VA Responses to NASEM Committee Report 11 10 20” document (herein referred to as the VA Response document) to be particularly useful in guiding its review of the revised monograph.

The committee recognizes the many challenges faced by the VA study team in the execution of a complex and important clinical trial, such as the logistics of standardizing the breeding, acquisition, training, and pairing of canines for Veterans across multiple study sites. The dog supply and quality issues are recognized as major challenges that impacted the timeliness of the study (and had additional impact on the data, analysis, and inference, in part due to withdrawals in participation). The committee also recognizes the time pressures associated with the study and the desire to disseminate the trial results to the respective stakeholders.

In accordance with the Statement of Task (see Box 1), the committee performed its review of the revised monograph by considering how well the revised monograph responded to the committee’s first report and whether the revised monograph is consistent “with accepted scientific principles and is suitable for publication.” The committee commends the VA study team on improving the monograph with respect to the majority of the suggestions and concerns conveyed in the committee’s first report. Notably, the committee notes that the VA accepted the recommendations to remove the health economics appendix and to add to the description of the interventions and the dog-related specifics. Information was added to the report regarding the breed and gender of the dogs, and requirements were listed such as the dogs

___________________

1 For the purposes of this letter report, the document titled “Monograph 1 (2nd draft) submitted to NASEM 11 10 20” will be referred to as the revised monograph.

Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×

needed to be well-socialized, attentive, and friendly toward people. In addition to the Canine Good Citizen training mentioned in the first draft, information on how the dogs were evaluated for distractibility and sensitivity to novel environments was added to the monograph. The visual presentation of the study results is a welcome addition to the revised monograph, as it could aid both the technical and nontechnical audiences in understanding the result for each outcome. Lastly, the interpretation of the study results is now more consistent with the hypotheses stated for evaluation. That is, the authors have clarified in the revised monograph that the study does not provide evidence that service dog placement improves overall disability or quality of life among Veterans with posttraumatic stress disorder (PTSD). However, the absence of clear evidence based on this individual study does not necessarily indicate the absence of effectiveness in improving overall functioning and reducing PTSD-related symptoms. Thorough reporting of the methods and analysis of this study could be instrumental in informing the design of future research on this important topic.

However, the committee notes below a few high-level concerns that remain to be addressed. Minor points that could be helpful for the VA study team to consider are also provided in Attachment E.

Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×

SPECIFICITY OF LANGUAGE AND INTERPRETATION OF RESULTS

While the interpretation of the results in the revised monograph is greatly improved, there are a few remaining concerns related to some of the language used that in some cases risks an over-interpretation of the study results.

The VA cover letter and the VA response document reference the fact that the monograph is intended to be informative to a wide audience that extends beyond scientific and clinical communities. Furthermore, the cover letter states that the monograph style and use of plain language (as compared to language used in a scientific manuscript) were done deliberately to facilitate understanding to a wider range of stakeholders. Recognizing that communicating the results of a complex clinical study to a lay audience is challenging, every effort must be made to prevent introducing inaccuracies via imprecise language, even if the differences appear subtle. More effort should be made to reduce the use of causal language when describing the results of the study; for example, replacing instances where the word “benefit” is used to describe “improvements” that occurred in both intervention groups and which might not be attributable to the interventions in this study. Imprecise terms can lead to subtle differences in interpretation by a lay audience relative to scientific readers. With regard to the interpretation of the study results, the committee’s first report discussed at length the importance of incorporating effect sizes, the associated confidence intervals, and clinically meaningful differences. The committee finds that the revised monograph does not address this concern sufficiently. The revised monograph still discusses improvements in disability at month 18, decreased suicidality from baseline to month 18, or improvement in anger reaction in the context of statistical significance, rather than clinical significance. Additionally, in the discussion section (starting on page 121), the revised monograph continues to hold to the statistically significant self-reported PTSD (PCL-5) finding as if it was a clinically significant improvement. The addition of one sentence stating that the PCL-5 finding is not demonstrable of a clinical advantage does not sufficiently offset this shortcoming. As PCL-5 is a secondary outcome, following at least three primary tests of association for the primary outcomes (World Health Organization Disability Assessment Schedule 2.0; Veterans RAND 12 Item Health Survey [VR-12] Physical Component Score, VR-12 Mental Component Score), the importance of the intervention effect on PCL-5 must be interpreted in the context of the magnitude needed for clinical importance. In short, a statistically significant difference is not evidence of a clinically significant difference. The VA author team should consider further revisions to clarify this for readers.

REITERATING THE IMPORTANCE OF THE INTENT TO TREAT ANALYSIS

The committee strongly recommends that the VA focus on the intent to treat (ITT) population in the analysis of all of the outcome measures, as the committee requested in its first report. In the VA study protocol and 2017 paper published by the VA study team (Saunders et al., 2017), the ITT population is specified indicating an understanding of the importance of this analysis. The authors should follow their protocol and provide this as a primary analysis. Without restating the case for the inclusion of the ITT analysis as discussed in the committee’s first report, the committee does want to emphasize why its inclusion is paramount compared to the per-protocol population, and the modified intent to treat discussed in the revised monograph.

When properly implemented and utilized, randomization produces unbiased estimates of treatment effects. However, post-randomization changes in the composition of those groups may introduce bias into the estimation of effects (a rule of thumb in many biostatistics courses is once randomized, always analyzed). In this study, removing participants from the analysis for reasons related to a mediating variable (an intermediate variable between the assignment and the outcomes) is especially concerning. The only unbiased way to estimate the causal effect of assignment to the service dog intervention is to compare the groups as they were randomized. Other analyses could be conducted to assess assumptions about missing data and the potential impacts of missing data; however, for a superiority trial, the primary analysis should include all participants in the groups to which they were

Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×

assigned. As noted in our initial recommendations, the ITT analysis that assumes outcome data are missing at random may be carried out with a longitudinal mixed effects model.

TRANSPARENCY

With respect to the committee’s first report, there was considerable attention paid to the Consolidated Standards of Reporting Trials 2010 (CONSORT 2010) guidelines (Moher et al., 2010) as a means of ensuring that the monograph contains the minimal information needed to fully understand the trial and its findings. Adherence to the guidelines also serves to ensure transparency in reporting. These guidelines, first published in 1996 and since endorsed by the International Committee of Medical Journal Editors and hundreds of journals, are a standard within the field of clinical trials and are not considered to be a matter of preference. Reporting all items required by the CONSORT 2010 guidelines is necessary to adhere to the standards and best practices in this field. It is worth noting that the committee does not intend this recommendation to require a re-write or re-structuring of the monograph, as long as all of the information required under the CONSORT 2010 guidelines is included within the monograph.

The committee recognizes and respects the need for study participant privacy, yet the points regarding data sharing raised in the committee’s first report remain. Properly de-identified or perturbed data could be posted publicly. Additionally, sharing the code used for statistical analysis would raise no ethical concerns and would enable others to vet and understand the models used. Lastly, publication of the full statistical analysis plan (including dated amendments) would better enable evaluation of the monograph within the scientific community. These points are raised in the spirit of considering the monograph as a stand-alone document that contains all of the needed information in one place. For example, there are details about the construction of the imputation model included in the VA response document that should be incorporated into the monograph, including a more complete description of the statistical model, the covariates included in the model, the number of iterations generated, and the software and version utilized.

RECOGINITION OF ONGOING TREATMENT

Lastly, the committee would like to see more clarification within the revised monograph that the changes reported in PTSD symptoms could be the result of ongoing PTSD treatment that is unrelated to the study intervention. Causal claims about “benefits” in both groups should be tempered by a discussion of the ongoing PTSD treatment, which should be reflected in the interpretation of the results. The revised monograph, the cover letter, and the VA response document are all very clear that it would be “unethical to ask participants to stop utilizing their existing PTSD therapies to allow a comparison of a dog only group (utilizing an unproven potential mitigation for PTSD)” (revised monograph, p. 28). However, the impact of ongoing treatment on the interpretation of the study should be more explicitly stated and accounted for within the text.

CONCLUSION

In accordance with the Statement of Task, the committee was tasked to review the revised monograph for how well it responded to the committee’s first report and for consistency with accepted scientific principles. This report transmits the committee’s findings that the monograph has been greatly improved, while noting the above outstanding concerns. In conclusion, the committee would like to once again thank the VA study team for its tremendous effort in revising the monograph.

Sincerely,

David B. Allison, Chair

Committee on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder

Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Page 1
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Page 2
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Page 3
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Page 4
Suggested Citation:"Letter Report." National Academies of Sciences, Engineering, and Medicine. 2021. Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder. Washington, DC: The National Academies Press. doi: 10.17226/26039.
×
Page 5
Next: Attachment A: Statement of Task »
Letter Report on Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In response to a request from the U.S. Department of Veterans Affairs, this publication reviews the Department of Veterans Affairs Monograph titled A Randomized Trial of Differential Effectiveness of Service Dog Pairing Versus Emotional Support Dog Pairing to Improve Quality of Life for Veterans with PTSD. Review of Department of Veterans Affairs Monograph on Potential Therapeutic Effects of Service and Emotional Support Dogs on Veterans with Post Traumatic Stress Disorder critiques the draft monograph and reviews it for consistency with accepted scientific principles and its suitability for publication.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!