Worldwide Education Statistics: Enhancing UNESCO's Role (1995)

Whereas UNESCO can still be described as “the world's premier source of international education statistics” (Puryear, 1995:81), BICSE's review of its education statistics program suggests that a decade of retrenchment and neglect has taken a serious toll. A review of the biennial reports of the Director General over the last two decades reveals a steady decline in the reported accomplishments of the Division of Statistics. While the division has been asked to become more responsive to a new population of statistics users (the World Bank, the other UN agencies, and personnel of research organizations and universities), its primary responsibility remains to the program sectors (especially education) within UNESCO and to assisting the member states. However, the biennial reports and interviews with current staff reveal that the actual operations are increasingly dominated by the servicing of the data base, preparation of the Statistical Yearbook, and a dramatically reduced servicing of member states' needs. Moreover, the quality of the data base itself is threatened by diminishing resources for data verification and documentation and by the virtual absence of the developmental activities that are essential to adapt to users' changing needs.

BICSE's assessment indicates that a decade of retrenchment has negatively affected UNESCO's statistical activities in a number of areas: the match between mission and functions, access to expert advice, data management, technological innovation, service to member states, quantity and comprehensiveness of publications; and inter- and intra-agency cooperation.

The emphasis given to capacity-building in member states and service to internal program sectors in the Division of Statistics mission statement is belied by the realities. As we have already discussed, data collection for and publication of the Statistical Yearbook overwhelm all other activities of the division. We discuss in more detail below the decline in service to member states and intraorganizational cooperation.

Lip service to the mission statement, however, is paid in terms of management decisions that do not accord with present realities. For example, after being moved to the Bureau of Studies, Programming and Evaluation, the division was restructured from five sections to two (ST/DEV and ST/ DAT). ST/DEV and ST/DAT were made approximately equal in size and in the balance between professional and clerical staff, despite the fact that the bulk of the division's work now centers on ST/DAT's responsibilities for data collection and dissemination. Moreover, there are insufficient travel and other support resources to allow the division to meet the goal of spending 50 percent of its efforts on work with member states.

The mission statement fails to accord with realities in another sense. It makes no mention of users beyond member states and internal program sectors, despite growing requests for assistance from such users. The division, for example, provides a copy of its total data base to the World Bank every six months; gives a copy of the data base on education in Latin American/Caribbean countries annually to the Inter-American Development Bank; prepares information on literacy trends and projections and selected education indicators for UNICEF, and in 1994 accelerated its annual primary school survey to meet UNICEF's needs; provides annual education indicators and selected tables to UNDP and the United Nations Statistical Division; and replies to numerous other requests for assistance.

Strained relations with two major nations (the United States and Great Britain) after their withdrawal from UNESCO in the mid-1980s have deprived the Division of Statistics of connections to some of the world's leading statistical professionals and agencies. Budgetary and institutional constraints have further hampered the division 's ability to seek guidance from education and statistical experts on conceptual, methodological, and policy developments that should be reflected in the statistical program. Instead of assembling its own chosen groups of advisers, appointed on the basis of their individual expertise, the division must generally rely on meetings resulting from general invitations to member states that wish and can

afford to participate. Countries select their own representatives, who may change from one meeting on a given subject to the next.

The 25th General Conference in 1989 recognized the need for the division to have a more appropriate advisory structure and passed a resolution that “invites the Director General to study the possibility of setting up a standing panel of experts representing both producers and users of statistical and other information on education, science, culture and communication, with a view to reviewing regularly the Organization 's data collection activities and advising the Office of Statistics. ” The 26th General Conference in 1991 came back to this issue, inviting the Director General to “set up a consultative committee (Category V) of experts . . ., serving in their personal capacity” (UNESCO, 1993:1-2). Category V committees, however, are financed by the regular budget of UNESCO, and insufficient funds were available for this purpose. So the two meetings of experts that were held were so-called Category IV meetings, with interested countries sending representatives of their own choosing and participating if they chose to bear the expense themselves. There have been no meetings since the second Category IV meeting in January 1993.

The internal logistics involved in the annual data collection cycle that supports production of the Statistical Yearbook are complex. As the resources of the Division of Statistics have declined and its technology has become outdated, production of the yearbook has driven out most of its other activities. Even so, data management problems remain.

The Statistical Yearbook includes statistics on general references (population and literacy figures), education, science and technology, culture and communication, printed matter, book production, newspapers and other periodicals, film and cinema, broadcasting, international trade in printed matter, and cultural heritage (museums). These statistics are obtained primarily from 15 paper questionnaires distributed early in each calendar year through the UNESCO National Commission in each country.¹

Since the national commission is most commonly located in each member state's education ministry, the return rate, timeliness, and accuracy are

highest for this section of the Statistical Yearbook. For example, over 200 countries provide some of the data for first-level education, but only 50 to 60 may provide detail on book production, science research and development, and cinema attendance.

Technology limitations (see the next section) and the use of paper questionnaires mean that review, revision, and input of data once they reach UNESCO are heavily dependent on manual processing and scrutiny. Five data clerks (three of whom specialize in education) manually transfer data from the paper surveys to printed tables. Quality control is limited and has declined as resources have decreased. The data clerks have the previous year's responses so that new entries can be checked for consistency, and row and column totals are automatically compared to ensure consistency of amounts within tables. Beyond this, however, there is no use of software-driven “edit checks” that could be used for data set consistency. (For example, no algorithms are used to verify that the number of students in one grade is less than or equal to the number of students in the previous grade in the prior year. Although immigration might explain an apparently anomalous result, the statistical algorithm would be a valuable way to signal what data need further verification.)

If the data clerks have time and other sources of information are available, they may check questionnaire results with individual country reports or data provided by other agencies; these latter sources are also used to fill in missing data items on the questionnaires. The data clerks have become familiar over time with the countries and the data items that are most problematic. Appropriately, they focus their attention on these. The same data clerks do the same countries and levels of education year after year. Although this does allow them to develop special expertise, no system has been created to allow this expertise to be recorded for the use of others. Paper records are kept for some of these transactions, particularly if correspondence is involved, but these records are kept in the files for each country's annual data. The documentation has gaps, and it is difficult to review the overall volume, types, and patterns of the changes being made to country-submitted data.

A decade ago, the division would send hundreds of letters to member states each year seeking clarifications and corrections in the data submitted. Even then, responses were often slow or absent, and further follow-up was frequently not possible. In succeeding years, the level of effort devoted by the division to clarification and correction of the initially submitted data has fallen off noticeably.

The requirement to publish a massive statistical volume annually (the Statistical Yearbook) has further narrowed the division's focus as staff resources have declined. The division estimates that it spends 40 person-

months a year on the production of the yearbook, over and above the effort required to maintain the underlying data base.

In its document reviews and staff and user interviews, BICSE discovered three important problems resulting from limited attention to data management. First, there is widespread skepticism within the international community of data users about the quality of the data published by UNESCO. Second, users and potential users complain about the difficulty of accessing the data (for reasons that will become clearer in the next section) and the lack of documentation to help them interpret the data. Third, a great deal of the division's knowledge about individual country circumstances and the idiosyncrasies of country data resides in the heads of staff members or in individual paper files rather than in systematic technical documentation. More than 20 of the current staff are over 50 years of age, however, and at least two data clerks are expected to retire within the next few years. Since UNESCO has a mandatory retirement age of 60 and permits early retirement at age 55,² a large portion of the division's experience and wisdom is in danger of being lost over the next decade.

The data management problems of the Division of Statistics are exacerbated by outdated computer support (hardware, software, and technical staff).

As the staff realize, the division's ability to perform the necessary tasks involved with data base access, maintenance, and quality control is limited by the fact that the data base currently resides on a mainframe computer (IBM 3081) that does not support software for efficient or widespread access to the data. Any queries, downloads, or changes in reports must be prepared by writing new code in COBOL or FORTRAN³. This process is time-consuming and inefficient compared with the use of contemporary data base software. This mode of access also requires a high level of technical skill and involves direct manipulation of the data base. Therefore the division allows only two staff with the necessary skills to have direct access, including entering new data and creating special data sets for analytical purposes. This seriously constrains the flow of work in the division and restricts its ability to respond to nonroutine requests.

The division is in the process of moving the data base to a PC/LAN platform, which will provide more efficient tools for access and maintenance. The division has acquired a high-capacity PC (486 DX2/66) that will be used as a dedicated server for the data base; it is currently used exclusively as a development platform for the Oracle data base system being designed and tested.

Other technological constraints within the division, however, threaten to limit full realization of the new platform for the data base. Existing hardware and software that support administrative and analytical functions already lag behind the division's needs, even before the new data base platform becomes operational.

Currently, the division operates in a LAN environment consisting of a Novell/Ethernet (10BaseT) LAN⁴ connecting approximately 40 of the division's 44 PCs. In addition to the new server for the data base, there is a low-capacity PC (386 DX/33) that functions as the LAN server. It supports word processing and some spreadsheet use throughout the division and is only partially adequate for most division needs. It has insufficient processing capacity to provide reasonable response times for LAN users or adequate disk space for large, complex applications. Moreover, it has a large number of low-capacity PCs connected to it. Of the division's 44 PCs, only 4 have 486 processors operating at 50 Mhz or above. Of the remainder, 19 are 486/33 Mhz and the remainder are 386 or below. In addition, 15 machines have less than 8 MB RAM, considered the minimum for efficient operations in a LAN/Windows environment. This inhibits the division's ability to move beyond low-demand activity (word processing and other standalone work); high-demand data base and work group activities would degrade response times and undercut the efficiency of the network.

In addition to the limitations imposed by hardware, staff capacity to respond to user requests is hampered by limited capacity to use the full range of applicable software tools. For example, although spreadsheets are potentially powerful tools for reorganizing data, performing some analyses, and preparing ad hoc tables and other kinds of presentations, the division

only recently acquired spreadsheet software. It had to organize and conduct its own training due to lack of availability of training resources from UNESCO's Division of Personnel. In an informal survey conducted as part of this report, all but one division staff member indicated a need for training in at least one software tool, and most expressed a desire to add one or more tools as well.

The division's mission statement identifies two primary audiences: member states and internal program sectors. Significant lip service is given within the organization to the importance of service to member states; and, as noted above, the division is formally structured with the goal of devoting 50 percent of staff and other resources in support of member states. In fact, however, the increasing preoccupation with production of the Statistical Yearbook and limited funds for travel and other nonstaff expenditures severely restrict the division's ability to work with member states. The major activity in support of these states comes not from the UNESCO regular budget, but from the extrabudgetary NESIS (National Education Statistical Information Systems) project, which is discussed more thoroughly below.

The combined loss of a professional statistician who was once a regular part of the staff at each regional office and retrenchment at headquarters have contributed to a shift in the nature of member state services. In the 1970s, states could expect more individual attention and a fairly rapid (two to four week) response by UNESCO officials to individual country needs. Now the emphasis is more on meetings and the preparation of training materials, although limited technical advice to individual member states continues.

The problem with the current array of services is less in their conception than in the level of effort the Division of Statistics is able to devote to them. The education indicators project, one of the special activities currently under way, is a good example of the mismatch.

In partnership with other organizations like the Asian Development Bank, UNESCO is attempting to improve the availability and quality of education indicators and their use as aids to decision making. The project has begun with a series of regional meetings in Africa, Asia and the Pacific, Latin America, the Arab states, and the Caribbean. It brings interested countries together on a regional basis to identify education indicators that each nation currently uses or wants to develop and offers assistance in refining and developing indicators and strengthening national capabilities for the systematic production, dissemination, and use of such indicators. It is designed to help countries improve their national data collection and processing. It also offers an opportunity for countries on a regional basis to

identify key indicators that they wish to develop jointly for purposes of international comparability. By taking a “bottom-up,” country-based, and policy-oriented approach to the identification and development of education indicators, it appears to provide important internal incentives for developing nations to improve their education statistical systems in order to generate the quality data needed to produce good indicators. The project is of great potential interest to, and is designed to contribute to, the indicators projects of other international organizations, such as UNDP, the World Bank, regional development banks, UNICEF, and UNFPA.

Nevertheless, the division estimates that it was able to devote only the equivalent of about 1.3 professional staff and about $117,700 (US) in nonstaff funds to this activity in the 1994-1995 biennium.

As the education indicators project suggests, efforts to provide services to member states often have the effect of mobilizing interest among both member states and donors, but the Division of Statistics is poorly positioned to capitalize on this interest. Not only is its ability to undertake capacity-building activities in individual nations quite limited, but it is also hampered severely in its efforts to serve as a catalyst and facilitator in partnership with other agencies.

As was noted earlier, the main service to member states currently provided is through the extrabudgetary NESIS project. The NESIS project developed from recommendations made by the Donors to African Education Working Group on Educational Statistics in 1990-1991. In October 1991 the project was initiated with funding from the Swedish International Development Authority. The NESIS project incorporates five components:

• a country diagnosis to identify statistical needs and determine the feasibility for intervention;

• the development of technical assistance modules in strategic problem areas;

• pilot projects in specific countries to test the technical assistance modules;

• preparation of the major implementation of the project based on the findings of the pilot projects; and

• production and dissemination of the technical assistance modules.

Phase one of the project (October 1991 to September 1993) focused on the design and pilot testing of the NESIS materials. The second phase (October 1993 to September 1995) has emphasized the pilot testing and revision of the project methodologies and materials. Phase three (to run from October 1995 to September 1996) will stress the dissemination of the technical assistance packages as part of national development projects.

This last point is very important. The response to the NESIS activities

has created expectations for service well beyond what the NESIS project and its host, the Division of Statistics, have the funds or staff to provide. A total of 35 countries (14 English-speaking and 21 French-and Portuguese-speaking) have expressed interest in the project. At present, seven have proceeded to the stage of pilot projects (Ethiopia, Mauritius, Zambia, and Zimbabwe from the English-speaking group and Benin, Guinea, and Mauritania). Many other countries would like to participate more fully, but resources (staff time and travel budgets) simply do not permit acceptance of more countries into the project.

The project is dealing with this issue of participation in two ways. First, the basic methodology of the project—the training modules—is designed to stress local capacity development without the expectation of significant external assistance by consultants or project staff. Second, improving management information systems is intended to take place as part of individual country projects, funded on an ad hoc basis by individual donors. This latter aspect of the NESIS program is pragmatic but still poses the question of whether the expectations raised by the project in certain countries may be unfulfilled because of delays or the unavailability of donor support. In any case, the NESIS project provides a model for future division efforts to provide assistance to member states, a point to which we return later.

There are four major forms of publication engaged in by the Division of Statistics: the Statistical Yearbook, the Statistical Reports and Studies, the Current Surveys in Research and Statistics, and the Statistical Issues series. The division also works with the Education Sector on preparation of the World Education Report and other publications (such as those related to the International Consultative Forum on Education for All). In these latter cases, the role of the division is usually restricted to calculation of specific indicators or statistical tables from its education data array.

An indication of the decline in the division's ability to fulfill its original analysis and dissemination function is its failure to maintain its various publication series. The Statistical Reports and Studies series began with four documents in 1981-1982. Since then, there have been only eight publications. The Current Surveys and Research in Statistics series (a highly regarded set of reports) has not appeared since 1993, whereas in earlier years three to four issues per year were not uncommon. Finally, the Statistical Issues series, which had five issues in 1991, had only three publications in 1994 and two scheduled for 1995. The decline in such dissemination activities diminishes the division's visibility and reflects the staff's

need to devote almost all their time to basic data collection functions rather than to analysis or interpretation.

Outside UNESCO, we have previously noted a number of contacts between the Division of Statistics and other UN and international agencies, with the division both providing routine data and responding to special requests. In addition, it works with OECD and Eurostat on joint questionnaires so that individual countries do not have to provide basic education statistics to three different agencies. Within UNESCO, the division 's chief collaborative endeavor is with the Education Sector, in particular through providing statistical data undergirding the development of education indicators that appear in the Education Sector's World Education Report.

In this area, our review suggests two things. On one hand, by the staff's own admission, the division has had less and less time available to work with outside agencies or with the Education Sector on special projects or to respond to ad hoc requests. Thus it is not surprising to find skepticism in other offices and organizations about whether the division is up to the task of providing the education data that these organizations will need in the future for their expanding projects and publications. On the other hand, some of the criticism directed at the Division of Statistics is unfair, in that it reflects worldwide deficiencies (well described in Puryear, 1995) that no single agency can overcome, particularly not an agency responsible for reporting on 200 nations at all stages of development and statistical sophistication. As Puryear notes (p. 80), “the principal problems afflicting global education statistics are at the national, rather than the international level.”

One area in which the division is seen by some to be abdicating its appropriate (and historic) leadership role is in revising ISCED. ISCED was a UNESCO creation, and its maintenance is one of the specific responsibilities that the organization has assigned to the Division of Statistics. ISCED has not been revised since its creation the mid-1970s. Although it is widely used, it is criticized for being (Puryear, 1995:85):

outdated, culturally bound and sometimes impossible to use. ISCED 's concepts and definitions are especially inappropriate for vocational and technical education, but need revision more generally to reflect the major changes that have occurred over the past two decades at several education levels.

The BICSE review made no attempt to judge the validity of these criticisms. We note, however, that they are not universal. Some opponents to revision insist that any new system will be no less arbitrary than the present one and that whatever gains are made in the system of classification of

levels and types of education will be offset by losses in continuity of the data series.

Although the desirability and feasibility of revising ISCED may be unclear, what is apparent is that the Division of Statistics is perceived as having moved slowly to address the concerns. Formal discussions about revising ISCED began at least as far back as 1992, when a “meeting of experts” was held. The division now expects that the ISCED revision will be completed during the 1996-1997 biennium. Whatever the substantive outcome, the pace of decision making has been frustratingly slow in the eyes of some users.