THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
BOARD ON AGRICULTURE AND NATURAL RESOURCES
December 10, 2008
Deputy Executive Director
Florida Department of Citrus
1115 East Memorial Blvd. Lakeland, Florida 33802
Dear Mr. Norberg:
Please find attached the results of the Committee on the Review of Research Proposals on Citrus Greening. This activity was supported by Contract No. 07-27 from the Florida Department of Citrus to the National Academy of Sciences. The review process was performed under the auspices of the National Research Council’s Board on Agriculture and Natural Resources.
Citrus Greening (also known as Huanglongbing or HLB) is a major threat to citrus production in Florida. In 2007, the Florida Citrus Industry Research Coordinating Council identified the disease as its number one priority problem and proposed an assessment on each box of citrus sold to be dedicated for research on ways to control the suspected causal agent—the bacterium, “Candidatus Liberibacter asiaticus” and its vector, the Asian citrus psyllid, Diaphorina citri. With revenues accrued from this assessment, the Florida Citrus Production Research Advisory Council (FCPRAC) and the Florida Department of Citrus formed a partnership and announced a research grants program to support innovative research leading to solutions for Citrus Greening and other citrus diseases.
At the request of the FCPRAC, and under contract with the Florida Department of Citrus, the National Research Council (NRC) agreed to organize an independent peer review of proposals submitted to the program in 2008. The grants program issued a Request for Proposals (RFP) in June of 2008 that attracted 236 pre-proposals, and ultimately, 205 final proposals, addressing a wide range of relevant research topics.
To conduct the independent review, the NRC appointed a nine-person Committee on the Review of Research Proposals on Citrus Greening, whose membership is listed at the end
of this letter. The Committee was assisted by 78 additional scientists, appointed to eight review panels, with collective expertise across the breadth of basic and applied expertise relevant to the problem of a vector-borne disease of this important agricultural crop. That expertise included plant pathology, plant science, molecular genetics and genomics, entomology, biotechnology and genetic transformation, horticulture, disease epidemiology, agricultural economics, agricultural engineering, and many specializations within those fields.
The organization of eight panels of reviewers (and the assignment of proposals to different panels) reflected the diversity of approaches encompassed within and among the 205 proposals to address the problem of HLB and other citrus diseases. Therefore, each review panel included a mixture of scientific expertise most appropriate for reviewing proposals grouped according to general research approach, as follows:
Epidemiology, Production Economics, Alternative Production Systems (19 proposals)
Insect Control (36 proposals)
Pathogen-Vector Relations and Disease Transmission (10 proposals)
Metabolomics, Proteomics, Transcriptomics, Host-Pathogen Interactions (24 proposals)
Genomics, Isolation, and Culture (26 proposals)
Pathogen and Disease Detection (25 proposals)
Disease Control (24 proposals)
Plant Transformation, Biotechnology, Screening for Disease Resistance (41 proposals)
Each of the eight panels was chaired by a member of the parent Committee. The chairman of the Committee served as an independent referee who was not involved in any of the panel reviews.
Roles of the Review Panels and the Committee
As noted earlier, the proposals were assigned to panels on the basis of predominant research approach, so some panels reviewed more proposals than others and, accordingly, some panels had more members than others. Each panel was tasked with conducting a review of each of the proposals assigned to it and for generating, for the purposes of assisting the Committee, a brief summary assessment of how well each proposal met the criteria of the program. In addition, to assist the Committee, each panel was asked to group those proposals considered to be worthy of funding into categories of high, medium, and low merit, and to the extent possible, to rank order the proposals within the groupings based on their evaluation. The panels were also asked to comment on the RFP and to give their sense of the quality of the process, and of the proposals themselves, in order to improve the assessment process for any future funding rounds.
The charge to the Committee was to: a) provide the review panels with guidance to ensure consistency in the review process; b) consider the recommendations of the panels as it examined the proposals; c) to evaluate the merits of each proposal relative to others similarly ranked across the panels; and, d) to develop a final list of proposals that would
be recommended to the Florida Department of Citrus and the FCPRAC for consideration of funding. The full, formal statement of task is attached as Appendix B.
Prior to their formal appointment to the Committee and/or the panels, all prospective members were screened for potential conflicts of interest, including for financial relationships with organizations and individuals involved in the review process. Each review panel and the Committee held a formal discussion of issues related to bias and conflict of interest, and each reviewed the composition of its membership relative to the expertise needed for the assessment of the proposals it had been given. None of the panel members were applicants to the grants program, although one was listed as a collaborator. In that case and a small number of other cases where institutional bias might occur, the proposal was assigned to a different panel or assigned to a different reviewer. In cases where a panel member had an institutional association with an applicant seeking a grant, the panel member did not participate in the discussion and ranking of that applicant’s proposal.
The Committee met by conference call in early September to discuss a common scoring and ranking process for the panels to follow, and met in-person on November 24–25, 2008 to conduct the final review. The review panels began working in earnest in mid-September (a few days after the September 5 deadline for receipt of final proposals) and each panel held one, 2-day meeting during the time-frame of early October to early November, 2008.
As the proposals were received and grouped according to panel, each proposal was assigned three principal reviewers from a given panel based on the disciplinary expertise of the panelists. These individuals were given the responsibility for providing individual written review comments and scores, leading the discussion of the proposal in the panel meeting, and preparing the panel summary evaluations following the panel discussion.
Criteria for Evaluation
Based on the review criteria described in the awards program RFP and the Committee’s direction to the panels on the relative weighting of the criteria, the reviewers used a worksheet to evaluate and score three general aspects of the proposals, as follows:
1. Relevance to the fundamental objectives of the awards program (20 points):
Relevance to focus areas (priority research topics listed in the RFP)
Likelihood that the proposed research can contribute significantly to the mitigation of Citrus Greening (HLB)
Clearly articulated and justified objectives for the research
2. Scientific quality of the proposed work (60 points), including:
Appropriateness and feasibility of the experimental approach and work plan (the likelihood of accomplishing research objectives)
Consistency of timelines and milestone with the nature of the project and proposed level of effort
Scientific soundness of the research approach
Overall strength of the rationale for pursuing the proposed approach
3. Capacity of the personnel and facilities; Appropriateness of Budget (20 points)
Backgrounds, expertise, experience, of the principal investigator, co-investigators, and collaborators
Appropriateness and completeness of the research team
Adequacy of the research facilities
Appropriateness of the budget request for the proposed task and clarity of the budget narrative
Prior to the panel meetings, the reviewers submitted their scores, which were used as the initial step in organizing the proposals for discussion. At the meeting, the panel members first examined the individual scores of each proposal to understand the basis of major differences in scoring by the reviewers and to make adjustments to the major groupings of the proposals. Following individual presentations by the three principal reviewers, the full panel engaged in a discussion of the merits of each proposal on the basis of scientific quality and relevance to solving the HLB problem, and placed the proposal into increasingly refined groupings and ultimately, a rank order according to priority (or merit) for consideration. At the end of the meeting, reviewers prepared short summary statements for each proposal. The summary statements and comments from the principal reviewers were provided to the parent Committee along with a memorandum from the chair of the panel describing the panel’s proposed rank ordering, suggestions for improving the RFP, and comments on issues related to the review process. (The summary statements are appended to this letter report as a non-public Appendix C).
The Committee received copies of all proposals and the panel review materials as they became available. Prior to it meeting at the end of November, the Committee chairman developed a normalized ranking of all 205 applications based on the number of proposals within a given group and its relative position in the rank order developed by the panel. This served as the starting point for discussion and re-ordering of the proposals.
Although the goal of the grants program was primarily to attract research proposals that addressed Citrus Greening, the RFP was clearly open to proposals on “other major citrus diseases” such as citrus canker. But because some aspects of the evaluation criteria were tied to a focus areas (priority research topics) related only to Citrus Greening, these proposals could not be fairly judged against the Citrus Greening proposals. Consequently, both the panels and the Committee evaluated these proposals as a separate group. In addition, during the panel reviews it was clear that a handful of proposals did not involve research but might be an important part of the infrastructure to support research and other activities related to sustaining the citrus industry; these included diagnostic services and the like. These also could not be judged using the same criteria as the research proposals, so they were separated from the others for separate discussion and commentary.
The proposals most highly ranked were discussed first, followed by those of medium or lower ranking. The chairperson of the respective panel assigned to review a given proposal gave an overview of its intended goals and his or her panel’s impression of its merits and shortcomings, according to the following questions:
What are the major scientific (and/or other) outcomes expected from the proposed project? What are the chances of achieving the expected outcomes (scientific merit)?
How are the expected outcomes to be applied in the control of disease in commercial citrus? What are the chances of applying the control measures successfully? Is it likely that the financial and regulatory cost of application would be acceptable (practical value)?
Is there significant overlap with existing or proposed projects? Is the overlap of value or is it a redundancy not likely to be beneficial?
Should a given proposal be found to be worthy of support, what parts of the proposal should be supported, and what are the recommended annual budgets and term of support?
Should the project be supported, what ancillary benefits might be expected for commercial citrus production or to the understanding of citrus in general?
As the discussions proceeded, the Committee adjusted the order of the proposals to reach a final ranking, including, in a few instances, reassignment of specific proposals to a different category—high, medium or low. Overlapping or similar proposals were compared to each other in order to determine whether the overlap might be beneficial and, if not, to identify the best team or best approach for obtaining important information. The Committee also identified instances when budget requests seemed excessive or insufficient and made note of redundant or less meritorious elements within proposals that should be eliminated. In some cases, the Committee concluded that, given the level of uncertainty or risk involved in a particular approach, the research should be supported initially for a period that would allow the investigator to satisfy reviewers that “proof of concept” had been established. Typically, such proposals were recommended for one year of funding.
RECOMMENDATIONS OF THE COMMITTEE
Proposals for Consideration
The Committee recommends to the Florida Department of Citrus and the FCPRAC for funding those proposals that ultimately were rated as being in the high or medium categories. Based on its evaluation, this includes 83 proposals, listed in rank order of merit in Appendix A according to whether they are Citrus Greening proposals, Other Citrus disease proposals, or Infrastructure. The list in Appendix A includes the last name of the principal investigator, the title of the proposal, the proposed duration of the project and the amount of the requested budget, the recommended duration and budget (as suggested by the committee), and brief comments from the committee where it is relevant to those individuals making funding decisions. Proposals that did not meet the committee threshold for overall quality were not included in Appendix A.
The Committee believes that the portfolio of proposals it has recommended represents a diversity of high-quality approaches to understanding and ultimately controlling the problem of Citrus Greening and other citrus diseases.
The responsibility for awarding up to $20.0 million (as indicated in the announcement of the RFP) lies with the Florida Department of Citrus and the FCPRAC. The cumulative total amount of the recommended budgets for the first year of proposals in Appendix A is equal to approximately $11.2 million. The funders should consider that if all of these research activities proceed successfully, it should presume to be committing itself to an outlay of another approximately $11 million one year from today and perhaps about the same in two years. Although the commitment to fund one year at a time is apparently imposed by legal structure of the box tax, the Committee is concerned that this uncertainty could ultimately interfere with the willingness and ability of the individual investigators to hire staff and purchase equipment.
For the current round of awards, the Committee suggests that the research sponsors ask potential grantees to submit revised budgets and more detailed budget justifications, perhaps along the lines of the format suggested in the subsequent section of this letter. The Committee also urges the sponsors to require that all pathogen and microbiome DNA and protein sequence data obtained as a result of this funding should be deposited in a public database as soon as is practical.
Other Considerations for the Future
Request more-detailed budget justifications and descriptions of the roles of the research team members.
The Florida Department of Citrus and FCPRAC are the bodies that will make decisions about funding, including partial funding of an application. A notable shortcoming of many of the proposals was the lack of detail provided in the budget justification. Many names appeared on proposals without an indication of what some of those individuals would be doing for the project. Similarly, some proposal budgets included services or consultants but sub-contractor letters were not included and consultants were not identified. The committee suggests that the applicants be given explicit instructions about what to include in a budget justification in the future. The applicant should be requested to provide, in addition to the budget justification narrative for each year, a table with columns corresponding to project objectives and rows corresponding to project cost components. Such tables, one for each year of support requested, would serve two major purposes: (1) to encourage the applicant to think through the overall project plan and (2) to provide reviewers with the connections between budget requests and the components of the proposed research and/or other activity. Specifically, column headings would be “Cost component,” “Requested dollar amount for component,” “Objective 1,” “Objective 2,”…etc. to the final objective. Row headings might be “Personnel” (with titles of appointees listed below) and the other budget categories of the current RFP. Cells would contain the percentage of funds corresponding to each objective and each cost component such that the sum for any cost component would be 100% of the budget amount for that component.
Require collaboration and limit the number of proposals per investigator
The Committee found significant overlap in some of the submissions from individuals who are very likely aware of what other laboratories are doing. Applicants need to be strongly encouraged to collaborate with others to take advantage of comparative strengths and to reduce redundancy.
Another issue was the number of proposals submitted by a single group of researchers. Sometimes these proposals were assigned to different panels because the major thrust of the proposal was different but the proposals would contain elements that were the same on more than one proposal. In some instances, the extent of overlap and its associated possible waste of resources (or benefit) became apparent only when the proposal reached the parent Committee. This is an unacceptable situation for the review panels. Applicants should be limited to a specified number of submissions. Alternatively, the sponsors should reserve the right, clearly spelled out in the RFP, to request the withdrawal of some proposals from individuals making multiple submissions, if such submissions are viewed as unnecessarily taxing the review process. Such a declaration in the RFP may be enough (and has been sufficient in another grants program) to encourage more thoughtful combinations of submissions.
Organize a pre-proposal seminar and an annual meeting of grant recipients
The Committee suggests that if there are to be future competitions, the sponsors might consider organizing a seminar to help prospective applicants understand the requirements of the RFP and the submission forms.
The Committee further, and strongly, recommends the organization of an annual meeting of grant recipients to catalyze synergies that could lead to new insights and to allow the community working on Citrus Greening to monitor progress of the research projects.
Comments on the Request for Proposals (RFP)
The Committee was asked to make suggestions for improving the RFP and the following comments are offered in that regard:
In addition to restructuring the budget page and provision of a more organized budget justification, as recommended above, the requirement to include a timeline and milestones needs to be made explicit in the RFP.
In some cases, the proposals were marked as “continuing” but no information or progress report from that research were included. Federal agencies often allow applicants to append 3–5 additional pages of progress reports or other data to the application. It would be preferable to be able to access progress reports on-line and have them referenced in the proposal.
The Committee hopes that a wider community of investigators will apply to the program in the future and encourages the dissemination of the RFP as broadly as possible. One idea for consideration is to create a separate review category for first-time applicants. Another possibility is to make the list of priorities slightly broader so that plant pathologists and vector biologists who work on other crops and plants would be attracted to offer their expertise to the problem.
This competition for innovative proposals to address Citrus Greening is a worthwhile activity and one that has challenged research institutions in Florida, nationally, and internationally to think creatively and purposefully. The Committee expects that the quality of proposals will improve over time, as experience is gained with each new round of the awards.
I sincerely appreciate the efforts of the panel reviewers. Their insights on the proposals were invaluable to the Committee during its discussions. They provided hundreds of individual review comments that informed our deliberations, and ultimately, the selection of the most meritorious proposals. The NRC staff, particularly Robin Schoen and Camilla Ables and their assistants, allowed us to have an organized, efficient and effective review process.
Members of our Committee hope that the Florida Department of Citrus and the FCPRAC will find our recommendations and comments on the budgets useful to the efforts of the citrus industry to eliminate the threat of Citrus Greening and other diseases.
University of California, Davis
Committee on the Review of Research Proposals on Citrus Greening