Improving Fish Stock Assessments (1998)

Marine fisheries provide a vital contribution to food supplies, employment, and culture worldwide. Therefore, matching fishing activities with natural fluctuations so as to avoid unsustainable harvests and population crashes is an important goal. In an ideal world, accurate and precise estimates of the abundance of fish stocks and their dynamics (how and why population levels change) would be available to set sustainable harvest levels to accommodate commercial and recreational demand. In reality, fishery management is based on imperfect estimation of the number, biomass, productivity, and age structure of fish populations and incomplete knowledge of population dynamics. The ocean is relatively opaque to light, and acoustic techniques of remote sensing are not yet sufficiently developed for general use in estimating fish populations. Thus, it is difficult to count fish through nondestructive means and fish usually must be caught to be counted, weighed, and measured. Standardized techniques have been developed to sample a relatively small proportion of fish from a population and to combine such data with commercial and recreational catch information to estimate population characteristics. These techniques yield stock assessments used by managers at state, regional, national, and international levels.

In addition to monitoring the abundance and productivity of exploited fish populations, stock assessments can provide a quantitative prediction of the consequences of possible alternative management actions. The mechanisms that cause fish populations to change are poorly understood but include environmental and ecosystem effects, interactions among multiple species, and effects of humans through harvesting, pollution, habitat disruption, and other factors. Without accurate stock assessments and their proper use in management, exploited fish populations can collapse, creating severe economic, social, and ecological problems. Therefore, ensuring that stock assessment research progresses and that operational stock assessments use the best techniques for a given stock are fundamental for ensuring the sustainability of commercial and recreational marine fisheries.

Stock assessment is a multistage process. Steps include (1) definition of the geographic and biological extent of the stock, (2) choice of data collection procedures and collection of data, (3) choice of an assessment model and its parameters and conduct of assessments, (4) specification of performance indicators and evaluation of alternative actions, and (5) presentation of results. This report concentrates on evaluating assessment models, with less extensive treatment of the other steps. Chapter 1 discusses these steps in greater detail. Techniques of stock assessment range from informal estimates to more sophisticated modeling approaches used to combine data of various types. Assessment models predict rates of change in biomass and productivity based on information about yield from fisheries and the rates at which fish enter the harvestable population (recruitment), grow in size, and exit the population (natural and fishing mortality).

Stock assessments for fish living in the U.S. exclusive economic zone (3 to 200 nautical miles from shore) and for some highly migratory species are conducted by scientists from the National Oceanic and Atmospheric Administration's (NOAA's) National Marine Fisheries Service (NMFS) and independent species group commissions (e.g., the International Pacific Halibut Commission and the Inter-American Tropical Tuna Commission). In addition, interstate fishery management commissions were created to facilitate the coordination of state assessment scientists in working with each other and with federal scientists to assess and manage stocks shared among states in their coastal waters (within 3 nautical miles from shore on open coasts, as well as bays and estuaries). These organizations include the Atlantic States, Gulf States, and Pacific States Marine Fisheries Commissions. Some states (e.g., Alaska, Oregon, and Florida) also perform assessments for fisheries conducted in their own state waters.

Fishery management organizations use the results of stock assessments to design and implement various controls for the total catch that can be removed from fish populations under their jurisdictions. Commercial catch can be managed by specifying the amount of harvesting allowed; the areas of fishing and times of the year that fishing can take place; the gear that can be used; minimum fish size limits; and in some cases, the amount of fish that any single fisher, community, company, or other entity can catch. Recreational fisheries more often impose minimum size limits, daily catch limits, seasons, and sometimes gear restrictions and requirements to release fish that are caught.

The National Research Council (NRC) Committee on Fish Stock Assessment Methods was formed in early 1996 to review existing stock assessment methods and to consider alternative approaches for the future. The committee's statement of task was two-fold:

1. Conduct a scientific review of stock assessment methods and models for marine fisheries management.
2. Compare models using actual and simulated data having a variety of characteristics, to test the sensitivity and robustness of the models to data quality and type.

As part of this study, the committee asked selected stock assessment scientists to conduct blind runs of simulated data sets using five different models. Models tested included a production model, a delay-difference model, and three age-structured methods (described in detail in Chapter 3). The goal of the simulation study was to evaluate the performance of stock assessment methods for simulated fish populations for which the true population parameters were known (to the committee, but not to the analysts) and some of the assumptions usually made in stock assessments were violated. One type of data set was typical of the catch biomass, age composition of the catch, and catch per unit effort (CPUE) that are obtained from commercial and recreational fisheries. The other type of data set was typical of that collected by fishery-independent surveys.

Each analyst was asked to evaluate five 30-year sets of simulated commercial and survey data, alone and in combination. The five data sets provided different combinations of parameters in terms of the following:

• Increasing or decreasing stock size over time (population trend)
• Constant versus changing age of fish caught (fishery selectivity) over time
• Accuracy of catch reported by fishers
• Ability of fishery and survey vessels to catch fish (fishery and survey catchability)

The analysts were given essential information about fish growth and maturity, the probability of mis-estimating fish ages, and selected information about the structure of the populations and the data. Analysts were not provided information about natural mortality, catchability, selectivity, recruitment, or the amount of underreporting (although they were warned that underreporting might have occurred).

In addition to the results of these basic analyses, (1) some analysts repeated their model runs with the true

average natural mortality (provided by the committee), (2) key management variables were calculated by analysts and the committee, and (3) retrospective analyses were conducted by the committee to determine the persistence of over- or underestimation of population parameters over time by the different models. Greater detail about the study process is given in Chapter 5 and Appendix E.

The committee focused its examination on the data that are used in assessments, model performance, use of harvest strategies, new assessment techniques, periodic review and quality control of assessments and assessment methods, and education and training of stock assessment scientists. The committee based its recommendations on the results of the simulations and on its collective experience. Caveats about how the analyses conducted for this study compare to actual stock assessments are given in Chapter 5. Accomplishing the recommendations of this report will require concerted and cooperative action by all interested parties (academic and government scientists, fishery managers, user groups, and environmental nongovernmental organizations) to improve the stock assessment process and products.

The committee concludes that stock assessments do not always provide enough information to evaluate data quality and to estimate model parameters, and it recommends a checklist that would promote more complete data collection for use in stock assessments. The results of the committee's simulations demonstrated that the availability of continuous sets of data collected by using standardized and calibrated methods is important for the use of existing stock assessment models. The best index of fish abundance is one for which extraneous influences (e.g., changes in gear and seasonal coverage, changes in fishers' behavior) can be controlled. The committee recommends that at least one reliable abundance index should be available for each significant stock. CPUE data from commercial fisheries, if not properly standardized, do not usually provide the most appropriate index. Likewise, CPUE data from recreational fisheries require standardization to serve as a good index of abundance.

Fishery-independent surveys offer the best opportunity for controlling sampling conditions over time and the best choice for achieving a reliable index if they are designed well with respect to location, timing, sampling gear, and other considerations of statistically valid survey design. NMFS should support the long-term collection of fishery-independent data, using either the NOAA fleet or calibrated independent vessels. Diminishing the quality of fishery-independent data by failing to modernize NOAA fishery research vessels or by changing sampling methods and gear without proper calibration could reduce the usefulness of existing and future data sets.

The simulation study demonstrated that assessments are sensitive to underlying structural features of fish stocks and fishery practices, such as natural mortality, age selectivity, catch reporting, and variations in these or other quantities. Auxiliary information in the form of indices or survey estimates of abundance, population structure information, and accurate estimates of other population parameters (e.g., natural or fishing mortality, growth, catchability) improves the accuracy of assessments.

Formally reviewed sampling protocols for collection of commercial fisheries statistics have not been implemented in many geographic regions. The lack of formalized, peer-reviewed data collection methods in commercial fisheries is problematic because bias and improper survey conduct may exist, with unknown impact on data reliability. Greater attention should be devoted to sampling design based on an understanding of the statistical properties of the estimators for catch at age and other factors. Sampling and subsequent analysis should also consider the issue of systematic biases that emerge with factors such as misreporting. Formalized sampling protocols have been developed for recreational fisheries in the form of the Marine Recreational Fisheries Statistics Survey (MRFSS). MRFSS data and methods, albeit imperfect, have undergone independent peer review, are readily available, and could serve as a model for commercial fisheries. The committee recommends that a standardized and formalized data collection protocol be established for commercial fisheries nationwide.

Both harvesting strategies and decision rules for regulatory actions have to be evaluated simultaneously to determine their combined ability to sustain stocks. Simulation models should be realistic and encompass a wide range of possible stock responses to management actions and natural fluctuations consistent with experience. The committee recommends that fish stock assessments present realistic measures of the uncertainty in model outputs whenever feasible. Although a simple model can be a useful management tool, more complex models are needed to better quantify the unknown aspects of the system and to address the long-term consequences of specific decision rules adequately. Retrospective analyses performed by the committee showed that persistent overor underestimation can occur over a number of years of assessment, regardless of which model is used. The committee recommends the use of Bayesian methods both for creating distributions of input variables and for evaluating alternative management policies. Other methods for including realistic levels of uncertainty in models also should be investigated.

In the simulations, model performance became erratic as more variability or errors were introduced to data sets. Newer modeling methods offer promise for reducing bias in key parameter estimates, although using mathematically sophisticated assessment models did not mitigate poor data quality. Different assessment models should be used to analyze the same data to help recognize poor data and to improve the quality of assessment results. Results from such comparisons can be used to direct survey programs to improve data quality and to assess the degree of improvement in data achieved over time. Greater attention should also be devoted to including independent estimates of natural mortality and its variability in assessment models. Further simulation work of this kind is also needed to determine whether the simulation results and the conclusions based on these results remain the same over multiple replications.

The committee believes that single-species assessments provide the best approach at present for assessing population parameters and providing short-term forecasting and management advice. Recent interest in bringing ecological and environmental considerations and multi-species interactions into stock assessments should be encouraged, but not at the expense of a reduction in the quality of stock assessments.

Although the committee did not evaluate alternative harvest strategies, it believes that assessment methods and harvest strategies should be evaluated together because harvest strategies can affect stock assessments and the uncertainty inherent in stock assessments should be reflected in harvest strategies. Despite the uncertainty in stock assessments, fishery scientists may be able to identify robust management measures that can at least prevent overfishing, even if they cannot optimize performance. Conservative management procedures include management tools specific to the species managed, such as minimum biomass levels, size limits, gear restrictions, and area closures (for sedentary species). Management procedures by which the allowable catch is set as a constant fraction of biomass (used for many U.S. fisheries) generally perform better than many alternative procedures. However, errors in implementation due to assessment uncertainties could result in substantial reductions in long-term average harvests in some years if biomass estimates are highly uncertain. Assessment methods and harvest strategies need to be evaluated simultaneously to determine their ability to achieve management goals. Application of risk-adjusted reference points (based on fishing mortality or biomass) would immediately lead to reduced total allowable catch and thus create an economic incentive for investment in improved data gathering and assessment procedures to reduce the coefficient of variation of biomass estimates.

There are at least four alternatives to harvesting a constant fraction of exploitable biomass that may result in levels of total mortality that are consistent with maintaining a fish stock. First, target fishing mortality can be reduced as a stock decreases in size to reduce risks. Second, a minimum biomass level can be established, below which fishing would be halted (this is done for some U.S. fisheries). Third, the size of fish captured can be increased by changing requirements for harvest gear. This restriction might allow smaller fish to escape and spawn, but could be ineffective if harvesters apply more effort to the larger fish. Finally, geographic areas can be closed to limit mortality for sedentary species if the distribution of organisms is well known and if the fishing

mortality in other areas is not increased. Area closures have been implemented or proposed for many fisheries worldwide in the form of marine reserves and sanctuaries.

NMFS and other organizations responsible for fisheries management should support the development of new techniques that can better accommodate incomplete and variable data and can account for the effects of environmental fluctuations on fisheries. Such techniques should allow the specification of uncertainty in key parameters (rather than assuming constant, known values), should be robust to measurement error, and should include the ability to show the risks associated with estimated uncertainty.

A few prominent recommendations for new approaches emerged from the study. Scientists that conduct stock assessments and organizations that depend on assessments should

• incorporate Bayesian methods and other techniques to include realistic uncertainty in stock assessment models;
• develop better assessment models for recreational fisheries and methods to evaluate the impacts of the quality of recreational data on stock assessments;
• account for effects of directional changes in environmental variables (e.g., those that would accompany climate change) in new models; and
• develop new means to estimate changes in average catchability, selectivity, and mortality over time, rather than assuming that these parameters remain constant.

The results from the simulation exercise should be sobering to scientists, managers, and the users of fishery resources. The majority of the estimates of exploitable biomass exceeded true values by more than 25%; assessments that used accurate abundance indices performed roughly twice as well as those that use faulty indices. A disturbing feature of the assessment methods is their tendency to lag in their detection of trends in the simulated population abundance over time. For example, some methods with some types of data consistently overestimate exploitable biomass during periods of decreasing simulated abundance and underestimate exploitable biomass during periods of increasing simulated abundance.

Although no stock assessment model was free from significant error in the simulations, it is also true that few of the models failed consistently. Hence, the message of this report is not that stock assessment models should not be used, but rather that data collection, stock assessment techniques, and management procedures need to be improved in terms of their ability to detect and respond to population declines. The simulation results and some actual fishery management examples suggest that overestimation of stock biomass and overfishing of a population can occur due to inaccurate stock assessments and that the overestimation can persist over time. The committee believes that the two most important management actions to mitigate this problem are: (1) to model and express uncertainty in stock assessments explicitly, and (2) to incorporate uncertainty explicitly into management actions such as harvesting strategies.

The absence of adequate data is the primary factor constraining accurate stock assessments. The differences between estimated and true values derived from the simulated data were most likely not introduced by any mistakes made by the analysts. Rather, the large differences that occurred under some scenarios were primarily the result of poor data and model misspecification stemming from incomplete knowledge of the true situation by the analysts. The surplus production and delay difference models did not include the ability to account for changes over time in key parameters for the simulated populations. The simulated data sets were better structured for analysis by age-structured methods; hence, these kinds of models performed better. When they did not perform well, it was generally because the models used biased information (e.g., the fishery CPUE index) or did not account for changes in selectivity and catchability over time. Had the analysts been told about these data features, it is likely that they could have compensated for them and obtained better assessments. Some of the newer models appear to be able to achieve such compensation through the introduction of process errors. Nevertheless, modeling will never be able to provide estimates that are as accurate as direct knowledge obtained by measurement and

experimentation. Thus, if future stock assessments are to avoid some of the past problems, management agencies must devote the necessary resources to monitor and investigate fish populations in a stable research environment that fosters creative approaches.

It is imperative that stock assessment procedures and results be understood better and trusted more by all stakeholders. One means to achieve such trust is to conduct independent peer review of fishery management methods and results including (1) the survey sampling methods used in data collection, (2) stock assessment procedures, and (3) risk assessment and management strategies. When applied properly to stock assessments, peer review yields an impartial evaluation of the quality of assessments as well as constructive suggestions for improvement. Such reviews are most beneficial when conducted periodically, for example, every 5 to 10 years, as new information and practices develop. In addition, a complete review of methods for collection of data from commercial fisheries should be conducted in the near future by an independent panel of experts, which could lead to the adoption of formal protocols.

Reduction in the supply of stock assessment scientists would endanger the conduct of fishery assessments by the federal government, interstate commissions, and international management organizations and would hinder progress in the development and implementation of new stock assessment methods. NMFS and other bodies that conduct and depend on fish stock assessments should cooperate to ensure a steady supply of well-trained stock assessment scientists by using mechanisms such as personnel exchanges among universities, government laboratories, and industry and by funding stock assessment research activities. The training of stock assessment scientists should endow them with skills in applied mathematics, fisheries biology, and oceanography. Education of fisheries scientists should be organized and executed in such a way that it complements and augments the NMFS research mission and leads to improved management strategies for fisheries in the future.