Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Measurement Error in Productivity Statistics PAUL HOLLAND Educational Testing Service and BENJAMIN KING University of Washington Chapter 4 of the Panel report reviews the methods and data sources used by the Bureau of Labor Statistics (B[S) to construct the various measures of output per hour that it publishes. In this paper we discuss a method of assessing the sources and the effects of measurement error in published productivity statistics. Final conclusions concerning the accuracy of current measures would require a study that is beyond the time and re- sources available to this Panel. In discussing a general analytical ap- proach, however, our aim is to delineate the problem, to illustrate by example the way in which the analysis can be carried out, and thus set the stage for further research into these issues. MEASUREMENT ERROR DEFINED The use of the term "measurement error" is explained by the following: A productivity measure that is produced by the BUS is a function of a set of quantities that can, in principle, be empirically determined. Calling the desired quantities z,, Z2, ~ Zm, denote the productivity measure M = Mazy, z 2, ~ Zm). This work is a collaborative effort and the order of authorship is alphabetical. 221
222 PAPERS In this paper we will avoid the fundamental conceptual question of whether the function M is the proper way to express productivity when the inputs to the function are error free. Perhaps the function should be of some other form with a different set of quantities as the argument, say, Navy, v:, · · ·, vet or Pews, we, · · ·, wp), but in the following we take the function M as a given, and we are interested in determining whether reported values of M are measuring what they are supposed to measure in terms of the particular formula that has been adopted. ~ If M deviates from its "true" value, it is because the inputs to M deviate from the true zoo, Z2, · · ·, Zm. We therefore define an observed value xi = zi + eizi i = 1, 2, · · ·, m, (1) where ei is a relative error of measurement, i.e., a relative deviation from the desired (correct, true) value Zi. What we observe each period and what gets reported is Maxi, X2, ~ Xm), not M(zi, Z2, ·. ~ Zm), and un- certainty in M is the result of uncertainty in the ei that causes the xj to differ from the Zi. Thus the ei and the xi can be looked upon as random variables. Sometimes the phrase "margin of error" is used in similar discussions. In the present setting, the standard deviation of the distribu- tion of ei would determine the margin of error in ei and hence the margin of error in the observed quantity, xi. In our analysis we shall assume that the distribution of ei does not depend on the magnitude of Zi. A more general approach might relax this assumption. In discussing measurement error we are classifying under one heading errors from several possible sources: (1) First there are frame deficiencies. For example, a component zi may denote hours worked in a particular sector of the economy, say, business establishments reporting to State employment security agencies. Suppose, however, that the list of estab- lishments to which reporting forms are sent is incomplete and thus certain establishments are not included in the coverage for that sector. The error of underreporting in this case is due to a frame deficiency. (2) If an establishment is covered, but for some reason does not report, and failure to report is not corrected by a follow-up procedu. e, then the resulting error is called an error of nonresponse. (3) Erroneous responses due to faulty record keeping, misunderstanding of questionnaire items, deliberate falsification, and the like are classified under the heading of response effects. (4) One may have perfect frame coverage and perfect response, and yet if, for reasons of economy, it is necessary to sample reporting units instead of taking a complete census, there will be errors ~ An example of the difference between M and some other form. N. is that between labor and total factor productivity.
Measurement Error in Productivity Statistics 223 due to sampling variability. (In most operations of this type, sampling error is the least important source of total error. As is well-known, information from samples is sometimes more accurate than total counts because of the possibility of using skilled interviewers instead of un- trained field workers and self-reporters. ~ (5) Certain observed com- ponents xi in the productivity measure may be the result of ad hoc estimation or judgment. An example is the use of data from trade sources because they are the only data that are economically feasible to obtain. Another example is the error resulting from the imputation of a deflator to an area of the economy for which prices are not available. Further classification is possible and may be useful for the purpose of detailed assessment of the errors that enter at each stage of the process. In this discussion we shall lump everything under the term measurement error, regardless of the exact reason for the error. We repeat, however, that errors due, for example, to failure to take quality changes in products into account do not concern us in this paper. They would be called conceptual or definitional errors resulting from the misspecification of the function M, rather than measurement errors, in our scheme. OVERALL ERROR IN THE PRODUCTIVITY MEASURE The exact value that the set of errors (e,, e2, · · ·, e,',) will assume at any point of observation in the future is uncertain. This uncertainty can be expressed in terms of a multivariate probability function for the errors. ei. The assessment of the exact form of the probability distribution is a complex subjective task for a single individual, and it might be difficult to attain unanimity of assessment for a group of concerned decision makers. The contemplation of such an assessment, however, is a neces- sary ingredient in an analysis of the overall error in the function M. If the observed M is written as M = Maxi, = M(`z ,, · · ·' X,,2 · ·, Zn~)rLl !) + E(z 1, · · - ~ znl; e l ~ evil)] ~(2) then the function E can be interpreted as the relative overall error in M. 2 In principle, the probability distribution of E can be derived from the probability distribution for the errors ei. 2 The symbol E used for overall relative error here should not be confused with the expectation operator for random variables.
224 PAPERS The problem of assessment of the probability function is made easier if the errors are assumed to be multivariate normal. In that case, their distribution can be completely specified in terms of a set of means (expected values) and variances and covariances. Although the judg- mental assessment of these (m2 + 3m)/2 means, variances, and co- variances is exceedingly difficult, the task can be simplified by the reasonable assumption that many of the pairwise correlations between errors from different sources are zero. After the assumption of normality and the specification of the parameters of the distribution of the ej, there are at least three reasonable lines of attack in deriving the distri- bution of the overall relative error, E. 1. Derivation of exact distribution for E via normal probability theory. 2. Anoroximate solutions using Taylor series expansion methods on the function E. 3. Building a computer model of the error structure of M and simula- tion of the distribution of E by Monte Carlo methods. Method 1, exact derivation, is possible in principle, but does not necessarily lead to a distribution that is easily analyzed or evaluated numerically. Method 2, Taylor series expansion, depends on the as- sumption that the components ei are generally close to their expected values, and then it only permits the approximation of the distribution of interest. Method 3, computer simulation, as is well known, is never a totally satisfactory substitute for the exact derivation of a distribution it is difficult to manage and interpret without a thorough scanning of the full ranges of the parameters of the components of the model. Because exact derivation is complicated in this case, simulation is useful for studying the effects of various assumptions concerning the com- ponent errors ei on the overall error E. As we shall see below, using simulation methods, it is possible to approximate the moments of the overall error distribution and to study the shape of the distribution of E as well. MEASUREMENT ERROR AND BIAS A normally distributed error of measurement has equal probability of falling above or below its expected value. Thus, if the mean of a particu- lar error ei is zero, we expect the observed value xi to underestimate, as well as overestimate, the true value zi some of the time. Sampling errors, in particular, would be thought of as having this form. On the
Measurement Error in Productivity Statistics 225 other hand, there are some components of error in productivity mea- surement that are thought to lead to more systematic upward or down- ward biases in the overall statistic. An example was discussed in Chapter 4, where it was stated that the data from the current employment sta- tistics (CES) program are for hours paid for, not for hours actually worked. Data on hours coming from the Current Population Survey (cPs), however, measure something closer to actual hours in the work- place. The bias resulting from systematic overestimation of hours worked in the CES source can be represented in the model by specifying an expected value greater than zero for the error in the CES data. If one believed that this bias were the principal source of error from the CES source, then the standard deviation for the error would be specified to be very small. Case H of the computer simulation reported below illustrates this point. AN ERROR MODEL FOR PRODUCTIVITY FOR THE PRIVATE BUSINESS SECTOR In Chapter 4 a detailed description is given for the construction of a productivity measure for the private business economy. It is shown that the data required for the numerator and denominator of the output to input ratio come from many different sources, each of which was de- signed for some other purpose, often of higher priority than that of productivity measurement. The errors coming from the various sources in the makeup of the ratio play the role of the ei in the functional representation above. In order to illustrate an analytical approach to the assessment of overall error, we shall, at the risk of some oversimplification, express the annual private business productivity ratio as a function of its major components. It was observed in Chapter 4 that, in the terminology of the Bureau of Economic Analysis (BEA), the numerator of the BES measure is equal to gross domestic business product in constant dollars less the "residual" and less the imputed value (in constant dollars) of the services of owner-occupied homes. 3 In a typical issue of the Survey of Current Business (e.g., Table 3 of October 1977) one can find a decomposition of gross domestic business product into the components nonfarm, farm, and residual. The constant- dollar values for nonfarm and farm business product in 1976 are dis 3The residual is the discrepancy in constant dollars between GNP measured as the sum of final products and GNP as gross product originating.
o · - so in is . · c) o ~4 - o in o 'e hi an - ~ c) o s~ CQ c~ . - c~ m Ct .> 50 ._, o ~: o ._ ._ o ~L O C) S~ Ct S: O ~L m ~ - ~ O E~ ~ C) o U) O ~ ~ . O - e ~ ~3 ~ ~, ~ C v C) ~ <~) ~ ~O ~ ~ ~ c: O ~ C) ~(;_) ~O (t - ~O ~C ~ tV $- Ct = o - 0 C~ ,_ O ~ . _ ._ ~ V, 0 = O r-] = r ~ ._ ,> O ~ _ _ ~O~ ~ O-O O ... . OOO O 11 11 11 11 11 _ ~0 ~ r~ . .... . . ~ -c ~ ~ ~ oo- ~ ~ ~ ~- - - o ~ . .. r~ ~r~ r~ 00 1a~ c) . ~3 ~ c' ~o C o o V~' Ce _ ~ ~·- ~L) O C ~ o o ~O, _ _ _ ~t_ V~ - O , _ ~ _ o o ~Z ~ ~ ~o Z ~ 226
Measurement Error in Productivity Statistics 227 played in Table 1. It can be seen that subtraction of the BEA estimate of the imputed rental value of owner-occupied homes yields the same total in constant dollars for 1976 that is shown in Chapter 4 of the Panel report in Table 4-2 for "BES numerator (private business sector)". Because of the equivalence between the income and the product sides of the national accounts, it is possible, for the annual measure, to decompose nonfarm business product into the major sources of informa- tion in terms of payments to the factors of production (see footnote 8 of Chapter 41. Thus, we have the secondary breakdown into (1) employee compensation, (2) profit-type return, (3) net interest, (4) indirect business taxes, etc., and (5) capital consumption allowances, with amounts and weights for 1976 shown in Table 1.4 The five categories for the income side, plus farm business, enable us to express the measured numerator of annual productivity for the private business sector as XN = ZN + ZN (W,e, + W2e2 + Whet + W4e4 + WseS + Wield, (3) where the ei are the relative errors of measurement coming from the various sources and the Wi are the weights shown in Table 1. Although these weights are estimated from data and may change over time, in our analysis we assume that they are constant. A more elaborate model would relax this assumption. XN is the observed total value of the numerator, and ZN is the unobserved or "true" value. In a similar manner, we can write the denominator of the productivity ratio as XD = Zl) + ZD~ Weed, + When + Wince I`' + We ~ e ~ I), (4) where ZD is the true value of total labor input and XD is the observed value. The components, 8-1 1, of the denominator are described in Table 2. Another source of possible error in the construction of the productivity measure is in the transformation of output in current dollars to an amount in constant dollars. Since it is not possible to deflate the com- ponents of national income individually, the total value in current dollars of the numerator is adjusted by applying an implicit deflator that is derived on the product side. The method of double deflation, 4The current-dollar figures shown in Table 1 have the amounts due to the rental value of owner-occupied homes subtracted out. These figures are unpublished and were pro- vided by BEA consultants to the Panel.
228 PAPERS TABLE 2 Decomposition of Total Hours for Private Business Sector (Denominator of Annual Productivity Measure) into Component Parts Component 1976 Hours Component (millions) Weight Data Source Current employment 122,093 We = 0.92 CES program statistics: All employees covered by unemploy ment insurance Farm employees; farm proprietors; farm un paid family workers; nonfarm proprietors; nonfarm unpaid family workers; private house holds 20,743 Wg = 0.16 cPs Government enterprise 3,156Woo = 0.02 Less not-for-profit institu- 13,425W'' = - 0.10 tions TOTAL 132.5671 .00 Estimates from BEA data on employment and cPs average weekly hours followed by calculation of the implicit deflator for gross product originat- ing is described in Chapter 4. The combination of price relatives for many individual products and materials and services used in production, weighting information from sEA input-output tables, and various other data that are required sug- gests a complicated error structure and resulting uncertainty about the true magnitude of the deflator. For our simplified model we define a single source of error for this factor: Xp=2p+Zpe7, (5) where Xp is the implicit deflator that is applied to XN the numerator value, and Zp is the unknown true value. 5 Finally, the three factors are combined to yield the productivity statistic M = XN/(XPXD). (6) 5To consider the deflation procedure in terms of a single factor, Xp is an approximation of the actual procedure but satisfactory for illustrative purposes.
MeasurementError in Productivity Statistics The relative error in measurement is E = EM(ZPZD)/ZN] -1, or equivalently, (1 + Woes + + W6e6) E= - (l + e7) (1 + Wses + + W~ei~) COMPUTER SIMULATION 229 (7) A computer program that simulates the distribution of overall relative error for the model discussed above is available from the authors. The program is written in BASIC language for the Dartmouth College time- sharing system (DTSS). The user specifies the vector of means, standard deviations, and the correlation matrix for the eleven errors, ei. Then the program generates a succession of pseudo-random error vectors from the multivariate normal distribution with the specified parameters. With each randomly generated vector of errors, the overall relative error is computed. After a sufficiently large number of trials, one will obtain a fairly accurate picture of the distribution of overall error E as a function of the param- eters of the component errors, ei. The program was run for a number of cases, each simulating 1,000 trials. We present these simulation results for illustration only. A more detailed presentation would give standard errors or confidence intervals for all of the quantities estimated by the simulation. The use of 1,000 trials made the simulation estimates of the means (i.e., values denoted below as x) accurate to three decimal places. The simulation estimates of the standard deviations (i.e., values denoted below as s) appear to be accurate to between two and three decimal places, but we have not made a detailed analysis of their accuracy. The results are as follows. CASE A For case A the means and correlations are all set to zero. All standard deviations are set at 0.01. The result is x= 0.000 s = 0.015. Comment This model assumes that the probability is approximately 0.95 that the error in each source lies within plus or minus two per
230 PAPERS centage points from the true value, and that, furthermore, the errors are mutually independent. The result shows that the overall error, although expected to be zero, will fall with probability about 0.95 between plus or minus three percentage points. In the general case of independence and constant error variance the standard deviation of the overall error is about 1.5 times the common value for component errors. CASE B Conditions for case B are the same as those for case A except that the correlation between en, employee compensation, and en, CES hours, is set at 0.99. The result is x= 0.000 S = 0.011. Comment Here en, and es are based on the same survey (for annual figures, a total enumeration of employees covered by State unemploy- ment insurance), and one might expect the errors from the same source to be highly correlated. The simulation, however, shows that this correla- tion has little effect on the variance of the overall error. This is some- what surprising since the respective component weights are high. The same result occurred when the correlation was increased to 0.9999. CASE C The means and correlations are zero. The common standard deviation is set at 0.01, except standard deviations for en, en, es, e' are all set at 0.0001. The result is x= 0.000 s = 0.010. Comment In this model the two major components in the numerator and denominator, respectively, are assumed to be almost error free (very small standard deviations), and yet the standard deviation for total error is reduced only to 0.010, the common value for the remaining com- ponent errors. CASE D The means and correlations are zero. Common standard deviation is 0.01, except for en, the error in the price deflator. Its standard deviation is set at 0.0001. The result is
Measurement Error in Productivity Statistics x= 0.000 s = 0.012. 231 Comment Reduction of the error variance in the deflator alone has little effect when results are compared with those of case A. CASE E Case E combines cases C and D with standard deviations for en, en, ea., e9, and en each set at 0.0001. The result is x= 0.000 s = 0.002. Comment In this model, with the standard deviations for the deflator and the major components of both numerator and denominator all near zero, the standard deviation of total error is considerably reduced. Prom cases A, C, D, and E we may conclude that reduction of uncertainty is required in both the implicit deflator and the major components before reduction of uncertainty in the overall productivity measure can occur. CASE F For case F the means are set at zero. Common standard deviation of errors is set at 0.01. Intercorrelations between pairs of errors 2-5 are set at 0.90; the same value is set for correlations for pairs of errors 9-11. The result is x= 0.000 s = 0.016. Comment Correlations between pairs of errors from {RS sources in the numerator are set at high positive value, 0.90. Similarly, the errors in the denominator related to the cPs are intercorrelated. The results, however, show no effect on overall error variance. This is due to the fact that the numerator and denominator remain independent. CASE G Case G is the same as case F. except that the correlation between e, and e ~ is also set at 0.90, introducing a correlation between output and input. The result is x = 0.000 s = 0.012.
232 PAPERS Comment There is some reduction in total error variance-parallel to case B. This is in agreement with our general conclusion that correla- tion between numerator and denominator corresponding to real world expectations has little overall effect. CASE H For case H the common standard deviation is set at 0.01. The same intercorrelations are used here as in case G. including 0.9 between en and en,. Means are all zero with the exception of the mean of en, set at 0.10 and the mean of e7 set at 0.05. The result is x= - 0.128 s = 0.009. Comment In Chapter 4 the discrepancy in CES employment data between hours paid for and hours at the workplace is discussed. In an attempt to make the model more realistic we introduce an upward bias of 0.10 (i.e., 10%) in CES hours. Similarly, in order to simulate the possibility that the price deflator is overadjusting for the effects of infla- tion, we set the mean error for e7 to 0.05. The result shows dramatically the effects on the overall productivity measure. It has a downward bias of 12.8 percent. With the BASIC program used in these simulation runs, it is possible to store the first 200 observations of the simulated overall error E in a file for subsequent analysis. As an example, we show the histogram and normal probability plot for the results of case A. Figures 1 and 2 indi- cate that the distribution of the overall error E is approximately normally distributed when the variance of the component errors is small. (In Figure 2 the proximity of the plotted points to the 45-degree line indi- cates conformance to normality. Measures of skewness, kurtosis, and studentized range are also in line with expectations under the hypothesis of normality. TAYLOR EXPANSION APPROXIMATION FOR OVERALL ERROR The formula given in (7) for the relative error of measurement for a productivity statistic may be expressed as E = E(U. V Y) = {~1 + U)/~1 + V)~1 + Yell -1 (8) The statistical plots shown in Figures l and 2 were produced with IDA (the Interactive Data Analysis programs) on the Dartmouth Time-Sharing System.
Measurement Error in Productivity Statistics ~ 1 5 UJ 3 LL ~ 10 11 LL o 5 m 6 3 1 o -2 233 20 _ . . Mean = -1 .1828E-03 · Std. Dev.= 1.4456E-02 . · ·. ·e ·. . · - - · · - ~ · -- . · - · ~ ~-- - ~-- · - ~ ~ ~-- -- · - ~-- ~--- ·.-- - ~-- - -- · - ~-- - · ·-~ · - ~-- - - - - - · - - - ..-- - - - - - - - ~- ·-.-.~----.--~--~-~- . .i----i.~-. t ~ ---1 .-.-i Sample Size= 200 l -6000. E-05 -3000. E-05 0000. E-04 ERROR 3000. E-05 6000. E-05 FIGURE 1 Frequency histogram of 200 observations of overall error. E. Mean=-1.1828E-03 X Std. Dev.= 1.4456E-02 1 X Skewness = -5.3452E-03 1 Kurtosis = -3.0572E-01 1 2 Studentized Range= 5.0307E+00 1 1 1 Sample Size = 200 2 3 2 _ _ 2 3- x ~1 1 1 1 11 1 x x 1 2 3 3 X 1 3 1 1 - . 1 6 2 2 6 2 4 · 6 · 4 5 1 2 5 1 9 · 1 2 · 1 NOTE: Frequencies Over 9 Indicated by X X X -3 -2 -1 0 1 2 3 FIGURE 2 Normal probability plot of 200 observations of overall error. E.
234 where PAPERS U = We, + + W6e6 V= e7 Y = Whet + · ~ ~ + We,. In the simulation study, the individual errors e = fee .. e,,) are specified to have a joint normal distribution with mean vector m = (m I, · · ·, m ,, ~ and covariance matrix C. In the various cases considered there, m and C were given particular values. In all of those cases the variances of the ei were assumed to be quite small, and it is under such conditions that the Taylor expansion' (or "delta method") can be expected to yield useful approximations to the distribution of E. First observe that U. V, and Y are linear functions of the vector of errors e. Define the (3 X 11) matrix: /w} . . . L= {0 W6 0 ·-- 0 1 O O Then we have the matrix equation o o W8 . . . / U\ V ~ =Le ~YJ . . . ... o o W., (9) (10) where e is the transpose of the vector of errors. It follows from standard theory for the multivariate normal distribution that the U. V, and Y are jointly normally distributed with mean vector t/ v ~ = Lm and covariance matrix (11) D =LCL'. (12) The Taylor expansion method proceeds by expanding the function E(U, V, Y) about the means u, v, end y to a first-order (linear) approxi 7See Rao (1965, p. 321) for a more detailed discussion of this method.
Measurement Error in Productivity Statistics 235 mation in (U. V, Y) and ignoring higher-order terms. Thus we have E(U, V, Y) = t(1 + u)/~1 + v) (1 + y)] ~-1 + ~ U-u)/~( 1 + v) ~ 1 + y)] -(V-v)~1 + u)/~1 + V)2~1 + y)] -(Y-y)~1 + U)/~1 + v)~1 +y)2] . . . (13) The Taylor expansion method approximates the distribution of E by a distribution with mean value given by E(u. v, y) = t(1 + u)/~1 + v) (1 + y)] ~- and variance given by the variance of the linear part of the expansion in (131. In the present case this is given by defining the (1 X 3) vector T = t1/~1 + v) (1 + y)], - (1 + u)/~1 + v)2~1 + y)], -(1 + U)/~1 + v) (1 + y)2] i. (14) Then the Taylor expansion approximation to the distribution of E has a variance equal to b2= TDT', (15) where D is the matrix given by (12~. To be technically precise, we must observe that with (U. V Y' jointly normal the moments of a ratio such as E; U. V, Y) do not exist. The assumption of normality for the error components, however, is not needed, and normal distributions would, of course, never be observed in the real world. Distributions that are symmetric (roughly' but truncated at some point in each tail will serve quite well to express one s probability beliefs concerning the errors of measurement. These are in fact the types of distributions that are used in the simulations above with ''pseudo- normal" variates. To illustrate these calculations, in Table 3 we summarize the values of b and of E(u, v. yJ for the cases given for the simulation and compare them to the corresponding simulation results, x and s. Fron~ Table 3 we see that the Taylor expansion method and the simulation results are in close agreement in every case. The strength of the Taylor expansion method is the ease with which it may be applied to this type of problem. The method, ho``ever is
236 PAPERS TABLE 3 Summary of Simulation and Taylor Expansion Results for the Eight Cases Considered in the Text Simulation x s E(u,vy) Taylor Expansion a b A0.000 0.015 0 0.015 B0.000 0.011 0 0.011 C0.000 0.010 0 0.010 D0.000 0.012 0 0.011 E0.000 0.002 0 0.002 F0.000 0.016 0 0.015 G0.000 0.012 0 0.012 H-0.128 0.009 - 0.128 0.010 UWe would like to thank David Saxe at Educational Testing Service for making these calculations. based on a linear approximation to a nonlinear function. In the present case, this approximation is satisfactory. If the variances of the ej were substantially larger, this would not necessarily be true. IMPLICATIONS FOR MEASURES OF PRODUCTIVITY CHANGE In this paper we have illustrated an approach to the assessment of the overall relative error for a measure of the level of productivity at a single time period. It is also possible to extend this approach to assess the overall error in the percentage change in a productivity measure over two time periods. Such an extension will require the specification of the joint multivariate distribution of the relative errors of the components in both time periods. While this is a more complicated case than those we have considered, it may, in principle, be studied by the same tech- n~ques. For example, if the productivity measures for successive time periods are unbiased and the relative errors are uncorrelated, and have the same standard deviations, then it can be shown that the error in the per- centage change (i.e., the deviation of the measured change from the true change) has a standard deviation that is roughly 1.5 times the standard deviations of the respective relative errors. In other words, if relative errors in the level of productivity in successive time periods are, in fact,
Measurement Error in Productivity Statistics 237 random (i.e., their direction cannot be predicted) then uncertainty about the changes in productivity is compounded. CONCLUSION We have presented this analysis in order to show a direction in which research into the problem of measurement error might proceed. Many questions remain to be answered, such as, Who is best qualified to assess the distributions of errors from the various sources?, What are the parameters of these distributions?, and When is the model sufficiently complex to be considered realistic? With respect to the subjective assess- ment of error, a precedent may be found in the pioneering research of Simon Kuznets (1954~. In Chapter 12 (part 3) of his three-volume work, Kuznets meticulously reports in 12 pages of tables the estimates of "margins of error" elicited from the three investigators (including him- self) who were responsible for the totals in various industrial divisions and income categories. It is interesting to note that all margins of error were raised by one-half because it was ". . . found that all three investi- gators tended to underestimate the errors attaching to the results of their labors." In more recent research into revisions in national income and product accounts, A. Young (1974) displays subjective rankings of the reliability of various components of GNP and national income. We see from these two previous studies that, at least, the idea of persuading experts to make subjective evaluations of errors in reported statistics is not foreign to the agencies involved. It is our recommendation that the Bureau of Labor Statistics and the Bureau of Economic Analysis explore further the possibility of using techniques similar to the technique in our simplified example in order to estimate the overall margin of error in productivity measurement. The routine reporting of an expected value of productivity along with its estimated standard error, instead of the currently published single number, should be seriously considered for the future. At the least, the agencies responsible for the construction of the various measures should consider carrying out a limited type of "worst case" sensitivity analysis, e.g., What are the highest and lowest possible values of pro- ductivity change this year if such and such a component is assumed to have X-percent relative error? Users of statistics often depend, without verification, on intuitive ideas of the effects of errors when a test of the implications of these errors on simple models can provide better insight into their actual effects. The statement, "Everything will wash out in the end" is often more hope than reality.
238 REFERENCES PAPERS Kuzuets, S. (1954) National Income and Its Composition 1919-1938. New York: Na- tional Bureau of Economic Research. Rao, C. R. (1965) Linear Statistical Inference and Its Applications. New York: John Wiley. Young, A. H. (1974) Reliability of the quarterly national income and product accounts of the U.S., 1947-71. Order no. COM-74-11538 (July). Springfield. Va.: National Technical Information Service.
