6
REPORTS FROM THE BREAKOUT SESSIONS1
The breakout sessions on the second day of the workshop enabled four working groups to draft a mock performance standard on a topic under the theme of Post Approval Monitoring (PAM) of ongoing research projects using laboratory animals, to better understand the process involved in its development and implementation. All workshop registrants were pre-assigned to a group to ensure diversity in perspectives and each group designated a rapporteur who described the performance standard, followed by the process the group used to develop its standard. Each group was assigned one of the following topics:
- Personnel training in animal handling and procedures
- Workplace safety
- Scientific flexibility, or study drift, in animal use protocols
- Perioperative surgical management
Under each topic, the working groups would address the following:
- Who are the stakeholders?
- Why develop this standard?
- What is the objective and/or desired outcome?
- What are the criteria for measurements?
- How should this performance standard be implemented?
- What would be the method of implementation and its schedule?
- What are the known and unknown limitations to this performance standard and what is the reporting responsibility?
_________________________
1 This section is based on the presentations during the workshop by the rapporteurs of the four ad hoc working groups. Statements attributed to the working groups have not been reviewed and do not necessarily reflect the views of the National Academies of Sciences, Engineering, and Medicine.
PERFORMANCE STANDARD FOR PERSONNEL TRAINING IN ANIMAL HANDLING AND PROCEDURES
Sandra Scherrer, Manager of the Preclinical PET Facility at the Feinstein Institute for Medical Research, began her report with a list of the relevant stakeholders, which included the IACUC, the Institutional Official (IO), the animal care staff, the research staff and the animals themselves. A performance standard for personnel training would ensure consistency in supporting good animal welfare practices across the board, as there is a need to document the abilities of staff and researchers to do good science.
The objectives and desired outcomes for this performance standard are consistency, competency, and proficiency, as well as high-quality training, said Scherrer. Measurement criteria included a post-procedural monitoring program for both animal care and research staff to establish a process for follow-up monitoring of animals after experiments are completed and for ensuring that staff understand that follow-up process.
This group, said Scherrer, wanted to create (a) consistent feedback loops among staff to enable effective communication, verification of competency and thorough documentation of training in a shareable format; (b) a prospective review of undesirable outcomes to prevent their occurrence; (c) an assessment plan to track personnel compliance; and (d) a plan for reassessment and retraining to correct deficits.
The working group, said Scherrer, proposed a system of self-review and self-reporting as a means of documenting training and competency. A didactic review, as part of a pre-training assessment would be conducted to identify individual needs. A mechanism for documenting the quality of the trainers’ training ability would help assess their improvement or decline over time, although progress/failure reports are not the best metric of a good training program.
The centerpiece of this performance standard, said Scherrer, would include a training program for trainers (“train-the-trainers”), a hands-on training program for staff and assessment of competency levels within trainee group.
The schedule and method of implementing this performance standard would depend on whether a program is big enough to have its own trainers or training coordinators and whether those people would need to be trained themselves. According to the working group, smaller facilities may not need to create their own training program, but larger institutions would need to estimate their resource needs before instituting this performance standard. A training program would be
composed of structured modules and both didactic and hands-on components that incorporate active learning.
Potential limitations include the need for significant resources in advance and for a team effort. Challenges may also arise from the dichotomy of research versus animal staff and the need to bridge cultural differences.
Developing the Performance Standard for Personnel Training in Animal Handling and Procedures
Bruce Kennedy, Compliance Associate at California State Polytechnic University, Pomona, explained that the group thought a culture of training, not just what was being taught, was important and that this message needed to come from the institution’s leadership. The working group recognized the minimum expectations for training in both the Guide and the AWR but thought that relevant language in these documents was not well-defined.
The group created a method to contribute ideas and share expectations, for example, to include evidence of what learning management was. The group voiced concern, said Kennedy, about competency and measures of ability to complete an activity or task after being trained, which led to the view that there needed to be competent trainers, competent trainees, and programs to train trainers.
The group also discussed rubrics and what it meant to have good surgical models or good cage changing processes for evaluating the competency of an individual once he or she has been trained. There was some discussion, Kennedy said, about outcomes and issues related to poorly trained individuals. The group also discussed ways to reinforce positive behavior and outcomes that reflected specific tasks pertaining to the care and use of laboratory animals.
The discussion kept returning, said Kennedy, to the question of assessing the adequacy of training while the group acknowledged that training is a lifelong endeavor requiring regular oversight. Inspections, accreditations, and semiannual reviews were stated as possible oversight mechanisms. The group also recognized training as a challenge for everyone involved in the care and use of animals. To avoid drift over time, the working group pointed to a need for concrete training programs adaptable to changing regulations, changing research projects, and changing institutional characteristics.
PERFORMANCE STANDARD FOR WORKPLACE SAFETY
Robert Dysko, Clinical Professor at the University of Michigan Medical School, presented the workplace safety performance standard, which would affect the IACUC, the IO, the animal care staff, the research staff, and the occupational and environmental health committees. A workplace safety performance standard would lead to regular review as part of the semi-annual review of an entire Animal Care Program. Desired outcomes would primarily include an increase in the percentage of staff enrolled in an occupational safety program; an assurance of training; and documentation of competency. Secondary outcomes would involve the establishment of minimum and acceptable levels of workplace injury and the identification and management of hazards at the laboratory site.
Measurement criteria, said Dysko, would be (a) adherence to occupational and environmental health standards and (b) tracking of incidents and exposures. The working group thought it would be important to provide a basic set of questions to be used by post-approval monitors to evaluate what was taking place in the laboratory without being led point by point through the approved protocol. Other criteria would include assessing staff’s knowledge of the occupational safety program; the consequences of not following the program’s rules; and a visual evaluation of the animal laboratory space.
The assessment plan would include timely reports of occupational safety-related incidents to the appropriate institutional committee, the IACUC, and the IO via the semi-annual report process. The report would include information gathered from all involved program components, such as environmental health, occupational safety, and the animal care program.
The action plan, explained Dysko, would include efforts to follow up on any reported incidents with retraining, lockout from the facility, or other actions to ensure the laboratory was compliant with all workplace safety codes. Assessment could be tiered to increase post-approval monitoring visits to areas in which there was an increased safety risk or elevated levels of non-compliance. The working group, said Dysko, thought it was important to identify individual stakeholders to be involved in the resolution and remediation process.
Regarding scheduling and the method of implementation, the working group suggested monthly and semi-annual notification of incidents and issues, Dysko said. The group noted that big changes at an institution, such as one company acquiring another, adding a new species
to an animal facility, or significant staff turnover might require additional training to address occupational safety and workplace safety compliance.
One important limitation was the (lack of) strength of an institution’s occupational and environmental health programs, said Dysko, while others included accuracy of reporting issues and exposures, and financial and institutional support.
Developing the Performance Standard on Workplace Safety
Dysko noted that the diversity of the group’s members played an important role in the discussions, especially during the brainstorming phase, so it would be worth bringing in people from outside an institution to provide different perspectives. As an example, Dysko noted that workshop speaker John Bryan, a member of the group, explained that aerial darting of wildlife from a helicopter was one of the easiest protocols to evaluate for occupational safety, something nobody else in the working group could grasp since helicopters are not common in most institutional protocols.
This working group, said Dysko, noted the following challenges:
- Accounting for the many ways in which institutions set up their workplace and environmental monitoring and enforcement programs
- Establishing protocols to enable the IACUC to monitor occupational safety concerns, normally the purview of other institutional entities
- Changing institutional culture regarding the importance of occupational safety and compliance.
PERFORMANCE STANDARD FOR SCIENTIFIC FLEXIBILITY OR STUDY DRIFT IN ANIMAL USE PROTOCOLS
Kent Lloyd, Professor and Head of the Mouse Biology program at the University of California, Davis, and Judy MacArthur Clark served as the rapporteurs for this working group. Lloyd noted that the word “drift” does not appear anywhere in the Guide, while the word “flexibility” appears only five times and never in the context of scientific research or progress. Thus, he said, it was first necessary to define terms, and the working group decided that while drift is a natural occurrence of science, it can cross over into non-compliance of an IACUC-approved protocol. From that starting point, Lloyd explained, the working group focused on protecting and promoting animal welfare while reducing administrative burden.
MacArthur Clark explained that drift occurs due to a tendency for investigators to go outside the terms of the IACUC-approved protocol because the approved authorizations are so tightly constructed. As a result, investigators either have to apply for approval of an amendment or they continue along, knowingly violating the approved protocol and reporting the deviation in the regular post-approval report. One way to resolve this problem would be to introduce more flexibility into the protocols, as long as animal welfare is not compromised. The resulting performance standard would be more flexible, allowing science to evolve while avoiding unnecessary bureaucracy and protecting animal welfare. The working group noted that culture change, in addition to a performance standard, would be necessary in both the IACUC and the investigators to truly realize the benefits of flexibility.
The desired outcome of this performance standard is to have appropriately authorized and reviewed research protocols with the IACUC recognizing the need for flexibility but also inflexibly protecting animal welfare. For example, said MacArthur Clark, if a study called for four blood draws over the course of an experiment, but might occasionally require six, the IACUC might decide there is a significant burden in the welfare of the animal from six blood draws and so it would not authorize the additional draws. However, if the IACUC takes the view that six blood draws would not be any more detrimental to the animal that four, it might authorize them with the proviso that the investigators should do the minimum number whenever possible because that would be less intrusive to the animal. An important component of this performance standard, then, would be to train the IACUC to think about flexibility in a new way and to train investigators to request realistic and reasonable flexibility in their protocols.
Regarding measurement criteria, MacArthur Clark said one approach would be to measure how many trivial amendments are granted with little consideration or debate by the IACUC. If this occurs frequently, it would indicate a problem with the initially authorized proposal. If the IACUC does not have to deliberate about an amendment it probably did not need to be an amendment in the first place and should have been put into the original proposal, the working group noted. Another measure would be to examine the number of essentially technical non-compliances that occur without endangering animal welfare. An example of a technical breach would be if a proposal called for collecting blood from an anesthetized animal via cardiac puncture but the investigators found it was easier to draw blood from the abdominal aorta. This change would make no difference to the animal but would be a breach in the protocol
nonetheless because that was not what was detailed in the proposal. Another measurement criterion, MacArthur Clark added, would be the number of interventions by the attending veterinarians.
The working group, MacArthur Clark reported, thought a survey, including nonscientific staff, would be an appropriate way of gathering information on how well the performance standard is being accepted and how well it is working. The working group further suggested that before implementing this performance standard, the IACUC would need to explain to the institution the rationale for this new approach and how it will continue to protect the welfare of animals used. Part of this launch program might include a pre-survey to understand the views of the larger community and collect feedback on the new standard, while training at the institutional level will be an important piece of the implementation.
An important limitation for this performance standard, said MacArthur Clark, would be if researchers start taking flexibility to mean the liberty to make changes to a protocol whenever they want. The working group noted investigators may start preparing excessively complex protocols covering every possible variation and the IACUC will need to push back on that type of proposal.
Post-approval monitoring and ongoing review, said MacArthur Clark, could be accomplished through repeated surveys that focus on numbers of non-compliances and similar measures. These surveys could also solicit more qualitative views on how well the performance standard is helping researchers deliver their science, making it easier for an animal care technician to protect the welfare of the animals, helping veterinarians feel that they are having an impact, and enabling IACUC members to feel they are spending their time doing something important instead of wasting time processing trivial amendments.
Similarly to the other working groups, Lloyd noted the importance of having individuals with diverse experiences contributing to the discussions. MacArthur Clark recounted that the working group saw post-approval monitoring as an opportunity not so much to check whether or not people had complied with their protocol, but rather an opportunity for learning. An IACUC can identify the lessons about flexibility from one study and apply them to other protocols. She reiterated that the group had difficulty working through what the performance standard should be because it may conflict with the engineering standard nature of a protocol. Thinking about performance, she added, provides opportunities
for communication both within the research group and with others in an institution about how responsible animal research is conducted.
One clear message from the group’s discussion, said MacArthur Clark, was there cannot be gray areas in terms of compliance with an IACUC-approved research protocol, for that creates a slippery slope of determining what amount of non-compliance is allowed. Thus it is important to maintain a black and white view of what a protocol permits and what it does not. This approach still allows the IACUC to determine that there were no welfare consequences of noncompliance. However, it is still important to send the message to the investigators that they failed to comply with the protocol. Deciding on this hardline approach, said MacArthur Clark, was important for the group before it could agree on the need to allow more flexibility within a protocol so investigators can be compliant when making sensible decisions about alternatives.
PERFORMANCE STANDARD FOR PERIOPERATIVE SURGICAL MANAGEMENT
Randall Nelson, Associate Vice Chancellor for Research and Professor of Anatomy and Neurobiology at the University of Tennessee Health Science Center, served as rapporteur for this working group, which thought there were several reasons for developing a performance standard for perioperative surgical management. The main reason is the lack of specificity in the Guide and the AWR with regard to key elements of perioperative management. As a result, he said, there is a need to provide the necessary details to ensure consistency and to provide a mechanism for protocol flexibility when addressing unexpected outcomes. These outcomes may or may not affect the animal but they probably affect the science, thus the working group thought it important to respond to both of those situations. The stakeholders for this performance standard would include the IACUC, the IO, the animal care staff, the research staff, and the animals.
The working group, said Nelson, identified three objectives for this performance standard: (a) ensure animals are handled appropriately to minimize pain and distress; (b) set consistent standards for investigator performance that reduce outcome variability and achieve planned protocol outcomes; and (c) assess the competency of the staff to ensure consistent and competent execution of the research plan as approved by the IACUC. This last objective would require education to ensure that staff have the appropriate skills to conduct a study.
Criteria to be measured, said Nelson, include personnel assessments designed to understand what staff members are doing as well as what their needs are to do their job correctly; species-specific concerns; equipment and space concerns; observational intervals and intensity; how well trained the monitors are in assessing these criteria; and a review of the experimental design by those doing the experiment and those monitoring the researchers. The working group also thought that since protocols should incorporate flexibility, it would be important to measure adherence to the ranges and options stated explicitly in the protocols. It would also be important to ensure that protocols are written broadly enough to account for known possible adverse outcomes and provide mechanisms to deal with them. By accomplishing the latter, the result is protocols that are not rate limiting but flexible enough to account for the possible adverse outcomes. Another important criterion to measure would be preoperative animal health to ensure the animal is in a reasonable condition for surgery and appropriately acclimated. In addition, the planned use and execution of analgesia and anesthesia should be assessed, Nelson stated.
In terms of implementation, Nelson reported that the working group thought monitoring surgical outcomes would identify problems and determine the procedures needing modification to ensure both animal welfare and good science. The working group suggested using a checklist to verify adherence to approved procedures and protocols, and one was supplied to the working group by an online participant listening to this working group’s discussions (Figure 6-1). Such a checklist could help the investigators know what the IACUC had approved and reduce the possibility of intentional or unintentional deviation from IACUC-approved activities. An audit process could be developed to include both planning and outcomes.
In terms of implementing this performance standard, Nelson said the working group wanted to make sure there was a way for checking appropriate post-operative conditions, such as during recovery, and to make sure that whoever was responsible for monitoring these animals did so at regular intervals to ensure minimization of pain and distress. The working group also wanted to ensure that investigators would be involved with this monitoring.
In thinking about limitations, Nelson said, there is an opportunity to do a trend analysis across an entire program to determine outcome patterns that may need to be addressed. The working group noted that post-approval monitoring, especially when it comes to perioperative management, can be extremely important in raising standards and
minimizing pain and distress, and it should involve not just those working in the laboratory, but also animal care staff, IACUC members, and members of the compliance office. In the case of protocol deviations that warrant reporting, there should be well-defined procedures for disseminating information internally and for reporting it externally.
FIGURE 6-1 Post-approval monitoring program checklist
SOURCE: Working Group 4 slide
Nelson concluded the report from this working group with a message sent by one of the online participants, who wrote: “The goals of post-approval monitoring are to ensure that the protocol is being carried out as the IACUC understood and approved it. The protocol itself must be the framework for the evaluation using the checklist as a tool. In addition, the monitor can also ensure the approved protocol procedures (in this case perioperative) also adhere to the AWR, the PHS Policy, and the Guide.”
Developing the Performance Standard for Perioperative Surgical Management
Nelson explained that this group first identified the relevant text in the Guide and AWR on perioperative surgical management and then the topics missing from these documents. This information was used to construct a framework. The group also wrestled with performance measures versus checklists. The participants, said Nelson, chose to look beyond what was not in the Guide and in the regulations, and instead used convention, best practice, and professional judgment to develop a performance standard.
Joseph Newsome, Clinical Director of the Division of Laboratory Animal Resources and Associate Professor of Pathology at the University of Pittsburgh, said one theme he picked up on that he had not heard before was the need for well-trained moderators and leaders who can work through the process to develop performance standards. Kennedy said this emphasizes the need to establish an institutional culture that makes those resources available and expects them to be used. MacArthur Clark remarked that while people with process mapping skills could be useful and accelerate the progress that an IACUC can make, a good committee can do good work without a professional facilitator. There is much to be gained, she stated, from letting discussion range over many ideas and not always stay focused on the task at hand.
Steven Niemi said the discussions from the first day of the workshop and from the working group activities highlight how hard it is to develop performance standards and how thoughtful and thorough people have to be to create good ones. Commenting on Newsome’s idea, Niemi recommended against relying on outside experts and said this is an activity that an institution needs to undertake itself, drawing on the expertise and years of experience that reside internally.
Donna Mathews Jarrell asked if any of the groups could elaborate on how difficult it was to determine which metrics would be needed to reflect desired outcomes. Dysko said that the workplace safety group had an easier time identifying metrics for post-approval monitoring, but a harder time thinking about what the IACUC or IO would need to see to know the post-approval monitoring process was working. Lloyd said the scientific flexibility working group was able to identify metrics once it agreed on the reason performance standards were needed, which is to protect and promote animal welfare while reducing administrative and bureaucratic burden. The training group, said Kennedy, easily identified metrics for quantifiable components of a protocol, but not metrics that could assess skill. Scherrer added that for training assessment the group agreed the size and scope of the facility or institution played a big role in determining the type of useable metrics. A smaller facility, for example, can more readily monitor training. As a final comment, MacArthur Clark said her group soon realized that depending on hard numbers was not going to provide all the needed metrics and that the qualitative information from surveys combined with quantitative data was the best approach.
COMPARING AND CONTRASTING THE WORKING GROUPS’ PERFORMANCE STANDARD DEVELOPMENT PROCESS
As a final piece of the working group exercise, Mary Ann Vasbinder was tasked by the workshop organizing committee to compare and contrast the process the four teams went through to develop a performance standard. She found it fascinating and even amazing that the groups could develop a set of performance standards that are inclusive and flexible and that stress the importance of professional judgment. She also noted the importance of a project management process starting with creating a team, setting out a business plan, engaging key stakeholders, and getting buy-in from the institution.
It was clear, she said, that diversity in creating performance standards is valuable because of the perspectives and range of knowledge that different voices bring to the process. The importance of engaging all key stakeholders was another point that each of the working groups made.
Communication and leadership were two other key features identified as essential for the success in creating and implementing a performance standard. Vasbinder said one huge benefit from creating a performance standard is that by engaging key stakeholders, the process starts changing culture within an organization. Instead of simply checking
boxes, the process requires thoughtful discussion and communication among the stakeholders, she said. Strong leadership committing to the time and effort needed to develop good performance standards also creates a culture that encourages engagement and communication.
With respect to the planning phase, the groups identified desired outcomes by involving each member of the groups. When it comes to executing performance standards, Vasbinder added, pilot studies can help, but the best approach may be to roll out the performance standard and see what happens.
There was a significant amount of discussion within the four working groups about what to measure, how to monitor the effects of a performance standard, and how challenging it can be to create appropriate and practical measurements. The working groups noted the importance of developing both quantitative and qualitative measures and the ease of measuring failure but the importance of measuring positive outcomes as well. Though the latter is admittedly more difficult, it is important to reward success and provide positive reinforcement. There was a good discussion, Vasbinder said, about absolutes and gray areas and the need to establish limits, which is also a part of monitoring the success of these programs.
The groups pointed out the pivotal role of the IACUC in providing leadership and direction for developing and applying performance standards. Not only does the IACUC have to approve performance standards, but it is the collective body that gives an institution the weight to administer and monitor them properly. Vasbinder said she was also struck by the integrative ideas for performance standards that were brought to the table. She reiterated the opportunity to change culture and the ways in which such a change can engage people and get them to participate more fully in the process of developing and implementing performance standards. As a final point, Vasbinder said the working group stated the most important outcome of any performance standard is the welfare of the animals.
Vasbinder then asked other workshop participants for their comments. Dysko said that he will take back to his institution the need to capture diversity in the teams convened to develop performance standards. Too often, he said, these teams are focused narrowly on a presupposed solution.
Malak Kotb said one of the helpful lessons she learned was to build some flexibility into the performance standard and to not shy away from trying to convince IACUC members of such need, so investigators can comply without holding back the science or creating artificial results.
Vasbinder then asked the panel of rapporteurs if they had ideas on how to encourage IACUC members, IO’s, and others, including veterinarians, to embrace the idea of flexibility. Dysko said flexibility starts with those creating protocols, adding that too often researchers are their own worst enemies when they add too much detail to a proposal.
Reflecting on his own institution, Dysko said what needs to happen is for the veterinarians and IACUC members who review proposals to determine those areas that could benefit from flexibility, convey that information back to the investigators, and get them to take some of the specifics out of their proposals. The challenge, he added, will be getting all of the reviewers to agree to this approach. Nelson wondered if the problem of limited flexibility rests with proposals being written without getting input from the people who see and care for these animals daily.
Kennedy asked Dysko if his IACUC had any facility managers or animal care technicians as voting members, as they may be the ones most likely to suggest where flexibility would be beneficial. Dysko responded there are not, but they do come to the IACUC meetings and are encouraged to participate in the discussion. Scherrer said the IACUC at her institution does not have anybody at her level as a member, which she and her animal care colleagues are trying to change. Input from the animal care and laboratory staff, said Scherrer, is missing in many proposals.
Lloyd noted he is fortunate that his staff includes trained veterinarians who have served on an IACUC and having their input on proposals as part of a team effort is invaluable. Nelson added this type of team approach should be continued after the proposal is approved, perhaps starting with briefing everyone involved in a project on what was approved and how to carry out the specifics of the proposal.
Kurtz said that he had advocated for and firmly believes that the most important people in an animal care and use program are the husbandry staff members, because nobody knows the animals better on a day-to-day basis than they do. It is important, he said, to not only engage them but to get their advice when writing proposals and making plans to enact performance standards. Norman Peterson said he believes part of the problem with proposals lies with investigators who do not understand they are allowed to write amendments with the proper justification and that IACUCs will consider those amendments. Carol Clarke responded she used to be an IACUC coordinator and her institutions held pre-review meetings with investigators to discuss those kinds of issues. What was missing from that process was a team approach in which the reviewers worked with the investigators to improve their proposals before formal submission to the IACUC. Kennedy added that combining a pre-review
meeting with a pre-launch meeting would be a great idea, particularly if it better engaged the animal care staff.
Kotb asked if it would be possible for the roundtable to generate scenarios and examples to teach investigators how to include flexibility in their proposals. Kennedy said the Collaborative Institutional Training Initiative group at the University of Miami has put together an animal care and use course that includes those kinds of case studies. Nelson, who helped write some of these case studies, explained they provide detailed explanations of the many choices investigators can make and the ramifications of those choices. Nelson said there is a need to educate IACUC members and the animal care staff, as well as investigators, about what is acceptable flexibility because they do not know how much flexibility is appropriate and allowable.
