2
How to Do a Knowledge Assessment
The implementation of the knowledge assessment methodology will be a collaborative effort among the World Bank, an independent implementing team recruited by the Bank, and local participants. World Bank staff stationed in the country will be counted upon to provide a briefing for participants on the economic and policy status of the country. In addition, individual members of the staff may serve as expert members of the team. The implementation team will not be fully formed until after the national symposium at which the sentinel enterprises are selected, because the team will include experts familiar with such enterprises and have access to global databases to establish appropriate benchmarks.
The local members of the team will include experts drawn from the academic and the business community. In the event that other sectors such as health, education, and agriculture are emphasized in the knowledge assessment, then experts from those areas must also be included. Inclusion of members drawn from those government agencies or ministries that have responsibilities in the areas relevant to the knowledge assessment merits careful, case-by-case consideration. Thus, representation from the ministries of industry, education, telecommunications, information, and trade should be based on compelling reasons to include certain individuals. The principle to be respected is to avoid confrontation and enhance cooperation. If it can be argued that the presence of entrepreneurs and ministry people at the same table will produce an alliance among them, their presence should be sought. If the possibility of confrontation would stifle frank discussion and exclusion would not bias government officials against the study, then it should be minimized.
For somewhat different reasons, it is recommended that World Bank staff participate but the Bank not carry out the knowledge assessment itself. As the premier development organization in the world, the World Bank has become associated with certain positions on issues that are relevant to the knowledge assessment. Questions of deregulation, protection, educational priorities, and others where the Bank has a well-known position may be subjects of recommendations arising from the knowledge assessment, and the control of the Bank, or its dominating presence on the team, may lessen the impact of the recommendations. Since the product of the knowledge assessment is purely advisory, Bank staff would have ample opportunity to challenge the recommended options in the future, leading the participants to refine and strengthen their arguments.
The Bank briefing for the implementation team will become an important part of the record of the knowledge assessment. It should cover the state of the economy, the infrastructure, macroeconomic policies of the government, human resources, and other topics that the Bank mission considers relevant.
NATIONAL SYMPOSIUM
Three important tasks must be carried out before the various teams assemble to carry out the knowledge assessment. These are:
-
Identify local stakeholders from the private and public sector who will participate actively in the knowledge assessment and, as a result of their participation, generate the momentum and constituency for change.
-
Select the sentinel enterprises that will illuminate the flows and the mechanisms for and extent of application of knowledge in the country.
-
Prepare a map of the National Knowledge System, providing in effect an organogram and flow diagram that can be used to pinpoint the barriers and high leverage points that will be the focus of the recommendations.
These three tasks must be well prepared in advance by the assessment team before they can be accomplished at a National Symposium. The symposium, possibly organized by the World Bank, will bring together entrepreneurs in knowledge-related industries, scientists and engineers, and government officials with responsibilities in the information or knowledge area, as appropriate. Members of the implementation team should lead the symposium to make certain that
participants get a clear idea of the purpose and process of the knowledge assessment. The participants will include those most knowledgeable in the country of the workings of the institutions that make up the National Knowledge System, and their contribution will be essential.
HOW TO CHART THE NATIONAL KNOWLEDGE SYSTEM
Preparation for elaboration of the National Knowledge System will begin at the national symposium. The starting point will be a template of the functions normally associated with a national knowledge system. This template, upon which the map will be based, must have been already developed from the pilot project described below and the accumulated experience of earlier knowledge assessments in other countries. A few examples of the type of functions that appear in the template, organized according to motivation, creation, access, assimilation, diffusion, and use, are given in table 1 . It will be the first task of the national symposium to organize and label these functions according to the agency, ministry, or private organizations that have responsibilities to perform them in the country. These will include the universities, government agencies, professional and industrial associations, private firms, non-government organizations, and others. The labeling can be carried out by the symposium participants in working groups, and the discussion and analysis of which agencies have which responsibilities will have a fruitful impact on later discussions of barriers.
Once the labels have been put in place, the flow of knowledge will be charted. Beginning with the flow of knowledge into the countries, it will include the sources of knowledge within the country and the knowledge users, and it will show the links among the agencies mapped into the National Knowledge System.
HOW TO DO THE VIRTUAL CASE STUDIES
A virtual case study is similar to preparing a feasibility study for a new enterprise or the expansion into a new process, product, or service by an existing one. All the considerations that pertain to a real enterprise are reviewed step by step for the hypothetical enterprise. However the objective is not to plan a real investment but to identify barriers that prevent such enterprises from coming into existence or that make them unattractive, or to identify new mechanisms that might facilitate them.
The success of the virtual case study depends greatly on the selection of the participants, their knowledge of the country and of the industry or sector that embraces the sentinel enterprise, their familiarity with economic, cultural, and intellectual conditions in the country, and the skill of the facilitators. An unknown in each country is the cultural factor that may or may not make it comfortable for participants to join in this kind of “game.” (An experiment in structured group brainstorming carried out at the Organization of American States some years ago was pronounced by some unhappy participants “very gringo.”) In some countries local participants may gladly volunteer for this service, realizing that it is in their best interest or the best interest of their country. In others, they may demand to be paid as consultants. Many businessmen and women may see it as a risky activity from which they may personally lose or gain face or advantage. In many countries, it will be best to carry out the virtual case studies in English; in others the local language will facilitate discussion. Experience, beginning with the pilot project, will determine how to maximize the yield from this exercise in each culture.
Selection of Sentinel Enterprises
The selection of the sentinel enterprises will normally take place at the national symposium, where local stakeholders and World Bank staff will take an active role. Two issues are most important. The aim is not to “pick winners” in economic competition, it is to eliminate barriers so that winners will be able to flourish. Therefore it is essential that the sentinel enterprises be selected with due regard to the national economic reality and for the priorities set forth in national development plans.
With this in mind, the sentinel enterprises are intended to expose the weaknesses of the National Knowledge System and the barriers to productive use of knowledge. Not all the barriers will be directly related to knowledge and information; high fees or other impediments to establishing enterprises of any sort, high import or export duties, or poor airport facilities would be detrimental to a knowledge-based enterprise as it would be to any other. Yet a judicious selection of sentinel enterprises will point a finger at the same key set of barriers from different perspectives, and provide both the rationale and the basis for evaluation of recommendations for removing such barriers.
If the industrial sector is the target of the knowledge assessment, then a sample of sentinel enterprises might include a sophisticated financial service company, a venture capital company, a computer software company, an engineering consulting company, a precision foundry, a plant to process a natural product, a food product company based on a local formula, and a company to publish technical books and journals. It must be stressed again that the aim is not to lay the groundwork for establishing such companies nor to encourage others to compete with existing companies. The only market research that will be done is to determine whether market research can be done, given the available knowledge resources. Similarly, the recruitment of staff and managers will be limited to determining whether appropriate staff and managers might be recruited in the country or from abroad. Any participant who is encouraged by the proceedings to start such an enterprise will be entirely on his or her own. A participant who sees the threat of competition from analysis
TABLE 1 Functions of a National Knowledge System
Motivation Functions: Formulation, monitoring, and review of policies and plans affecting national knowledge activities; Allocation of resources to science and technology; Preparation of national development plan; Establishment of laws and regulations affecting communications and trade; Creation of incentive schemes to stimulate knowledge application and other technical activities; Technological forecasting and assessing the likely directions of new knowledge and technical change; Regulation of the stock market and other investment systems; Use of government's purchasing power as a stimulus to knowledge-based enterprises; Stimulation of public interest in and support of learning and innovation; Creation Functions: Formulate policy and set priorities for research and development funding; Carry out basic and applied research and development; Operate major national facilities for research and scientific training; Provide awards and recognition for successful original research; Monitor and evaluate the quality of research and development; Access Functions: Creation and operation of a national system for the recording and protection of intellectual property; Provision of linkages to regional and international information sources; Establishment, operation and maintenance of information services (including libraries, data bases, statistical services, system of indicators, communications systems); Provide up-to-date knowledge on advances in capital goods and process technologies; Provide opportunities for foreign study for public and private sector employees; Publish statistical indicators on science and technology; Publish register of laws and regulations; Assimilation Functions: Establish curricula and standards for science and engineering education; Provide programs and facilities for vocational and on-the-job training; Train engineers and researchers; Operate technological observatories abroad to advise firms on opportunities and threats; Diffusion Functions: Maintain a national system for metrology, standardization, calibration, information, and statistics; Operate technical services (e.g. metrology, standardization, calibration, information, statistics); Publish technical journals and newsletters; Organize scientific symposia and seminars; Provide technical consulting services; Evaluate and diffuse best practice technologies; Use Functions: Provide assistance in negotiation for technology transfer, licensing, and joint ventures; Promote innovative goods, processes and services embodying knowledge; Carry out export market research for local firms; Provide technical consulting service on quality control; Provide quality testing service accredited in major foreign markets; Provide advice to firms on opportunities related to GATT and other international marketing agreements. |
of a sentinel enterprise that is close to her own business must realize that she herself would be in the best position to take advantage of the presumed opportunity, but she must be encouraged to share her knowledge in order to benefit from the knowledge of the other experts.
Sample Sentinel Enterprise
For the sake of example, we might imagine one sentinel enterprise to be a software engineering firm. This type of firm would enable other firms to obtain new software to enhance their access, speed, agility, and quality management for a global market. In particular it might be able to produce sophisticated software in the local language that would increase its acceptability and ease of use for local clients.
The following types of individuals might participate in the virtual case study:
-
a software engineer from an international computer company
-
a computer or software engineer from a local company
-
a major software user
-
a professor of computer science at the national university
-
a local computer dealer
-
a representative of an industry group or association, users of computers
-
a representative of the finance community
-
a local patent or regulatory lawyer
-
members of the knowledge assessment team, with previous experience of virtual case studies.
-
an experienced facilitator.
The issues raised for discussion will include:
-
preparedness of persons or firms for starting such an enterprise
-
access to capital
-
availability of communications infrastructure
-
access to market information
-
consumer receptiveness to product or service
-
right to lease premises
-
procedures and permits required to launch a new enterprise
-
technical assistance on business practices
-
access to information on business software
-
technical assistance for selection of equipment
-
right to import equipment and software
-
availability of skilled labor
-
training opportunities
-
information on best practices
-
environmental protection, as appropriate
-
information and training on quality management
-
regulation on health and safety
-
opportunities for joint ventures, and permits required to launch them.
The exercise should be carried out in about a day.
If well chosen, the issues raised here will be similar to those for the next sentinel enterprise selected, which might be a formula-based soft drink company. The selection of two or more concrete but very different industries allows different elements of the industrial sector to become engaged and reinforces the generality of the recommendations.
It is essential that the participants be open and frank. Barriers will be recognized only when statements like: “that agency is very inefficient” or “bribes must be paid to complete that transaction” are heard and noted. Some of the issues will not be knowledge-related, but where knowledge is not the limiting factor, other types of barriers must be removed before knowledge can be effective.
Baseline and Benchmarks
For each sentinel enterprise, if similar enterprises already exist in the country, baseline data will be gathered to form a basis for later evaluation. This information will be gathered partly from the local participants in the virtual case studies, and partly from the data and interviews. It will include average and range of firm size, capitalization, technology used, health and safety standards, quality measures, skill level of employees, and market size.
Benchmarks are provided by the international participants. These are based on information obtainable from international, industry-wide data bases containing industry averages and best practices, indexed by country. The particular metrics used should correspond to the baseline data, if possible, to facilitate direct comparison for evaluation purposes.
Here is a rather simple example. One of the functions in the template might be to license the importation of equipment. The agency responsible is, say, the Customs Bureau, and this appears on the National Knowledge System map. Links in one direction are to the Ministry of Finance, and in the other to the industrial sector. For the case imagined above of a software engineering company, the “equipment” might be a sophisticated software package used for design of applications. The virtual case study will show that the relevant metrics are the time required to obtain the import license and the fees involved. The baseline is the current time and fee in the country. The best practice might be the time and fees charged in Singapore, or the maximum that a global firm such as IBM or AT&T could tolerate to consider a contract for design of software with such a company.
COLLECTION OF DATA
The virtual case studies will be an effective tool for probing the National Knowledge System, but at best they must be treated as revealing, albeit fictitious, anecdotes. The issues raised must be confirmed by valid data before recommendations can be made. For this reason, the data collection and interviews should be carried out after the virtual case studies
are completed. The barriers identified by the virtual case studies will guide the selection of statistical data and provide the issues for the interview guides. Specifically, the barriers should be raised in the interviews in order to confirm the conjectures of the case study teams and to gather ideas for lifting barriers, building bridges, and remedial action.
The following quantitative indicators are intended to provide an example of knowledge available in the country and in international databases that will be relevant and available. Similarly the issues suggested below could be included in the discussions with individual entrepreneurs, scientists and engineers, government officials, and other knowledge workers. The pilot project will produce a more complete list of indicators with general sources and interview guides based on the virtual case studies.
Sample indicators and issues for the interview guides are listed according to the element of the National Knowledge System.
Motivation
The use of knowledge for productive development depends on the motivation and example provided by the leadership of the country, the government, and the productive sectors, and the incentives, financial and otherwise, for its application. Also included are enabling conditions like physical infrastructure and laws and regulations.
Sample indicators
financial structure
taxes and duties
cost of capital
cost of labor
number of exporting firms
venture capital sources and availability
limits on repatriation of earnings
infrastructure
power
roads
telecommunications
transport
postal services
laws and regulations
local content requirements
laws governing foreign direct investment
laws governing importation of technology
Sample interview issues
leadership
political stability
economic stability
trust in institutions
pent-up demand
political climate toward innovation
Creation
Creation of knowledge is the essence of the research and development system. However in most developing countries scientific resources and manpower are limited and not competitive with the best laboratories in the advanced countries, except on purely local scientific issues. Therefore institutions dedicated to the creation of knowledge should be linked to those providing access to knowledge for best allocation of resources.
Sample indicators
input indicators
number, quality of research facilities
number, quality of research graduate programs
R&D budget, by field and institution
output indicators
published papers; patents
collaboration with foreign scientists and engineers
Sample interview issues
rank in field of key research groups
contribution of research and development to productivity
adequacy of resources and manpower
linkage between university labs and industry
Access
Motivation to engage in knowledge-based productive activity will be in vain if there are not sources of information and of knowledge available within the country. The interesting indicators will relate to the institutions that provide the knowledge. These will describe sources of both embodied and codified knowledge, because their knowledge stock must be measured in learned people as well as books and products.
Sample indicators
institutions
number, quality of research universities
information services
existence of data bases
numbers of journals, newsletters, seminars
global networking
number of Internet users, servers
companies using Internet
on-line information services
PCs per capita
telephones or telephone lines per capita
membership in professional societies
numbers of joint ventures or amount of direct foreign investment
movement of peoples
Sample interview issues
political openness
access to government information sources
free press
advertising
legal structure relating to knowledge
Assimilation
Information is a tool, one that takes a trained worker to use effectively. The workers must know how to use the tool, and they must have it available precisely when it is needed. Information and knowledge may be accessible in a country, but they must be understood to be used. The question of assimilation involves human capacity, the availability of skilled and motivated people.
Sample indicators
public and private expenditures on R&D
workforce characteristics
natural scientists and engineers
technicians (bachelors and two-year college)
where graduates are employed
educational structure
literacy rate
fraction of students that attain various educational level, males and females
graduates in selected fields
foreign graduates, profile and return rate
number of research universities
computer education
continuing education; training programs
education of managers
Sample interview issues
business environment
preparedness of university graduates for industry
influence of Internet and other networks
antitrust policies.
Diffusion
Diffusion will depend upon communication links within each sector, and between these and the universities and research laboratories. Mobility of workers and existence of trade and technical publications contribute to the diffusion of knowledge.
Sample indicators
mobility of labor and management
number and type of industrial associations
number and type of knowledge parks and other sharing facilities
benefit plans that encourage or discourage hijacking of trained labor
Sample interview issues
inter-firm linkages
“communicativeness” and networking
attitudes toward collaboration
role of associations in spread of knowledge
use of trade shows
influence of capital goods industry
other mechanisms that spread knowledge
Use
Use is the concrete result of motivation, access, and assimilation and diffusion of knowledge. Data gathered in this regard will show the success or failure rate of knowledge-based enterprises.
Sample indicators
industry structure
patent and licensing data
number of knowledge-based enterprises
size distribution
local and foreign
imports and exports of technological goods
Sample interview issues
ease of establishing knowledge-based enterprises
consumer acceptance of innovation and new products
factors contributing to success or failure of knowledgebased enterprises
experience of best cases
Statistical Sources
Quantitative information will come largely from national sources, government agencies or private associations. Some international agencies have data for certain groups of countries.
Bureau of statistics
industrial surveys, educational surveys
Science and technology indicator reports
UNESCO, OECD
Industry associations
Interview Sources
The qualitative information will come from public officials, scientists and engineers, business men and women, and other producers. The selection should be focused around the
institutions identified as playing a role in the National Knowledge System.
Political leadership
Social and religious leadership
Information industry
Health providers
Universities
Chamber of commerce
Industry associations
Engineering associations
Professional associations
CEOs, large, global, and small and medium enterprises
HOW ARE RECOMMENDATIONS FORMULATED?
Recommendations will be aimed at establishing sustainable ways to maintain development. That is, permanently removing barriers related to knowledge or instituting incentives for productive use of knowledge will allow a continuous process of establishing knowledge-based enterprises, competition, incremental innovations, and continuing social and economic development.
Following the themes of the knowledge assessment, recommendations will consist of positive steps to improve motivation, creation, access, assimilation, diffusion, and use. Some of the recommendations may be more general, identifying barriers that are not specific to knowledge-based enterprises, such as regulation, energy, and infrastructure. These must be included, because it may be that knowledge-related motivation, access, assimilation, and diffusion are not the limiting factors, and one can not expect to see a change if the other issues are not dealt with.
Recommendations should arise from knowledge assessment in a systematic way. The elaboration of the National Knowledge System, based on the template developed during the pilot project, will reveal institutional weaknesses on a coarse-grained scale. Lack of a government office to carry out an important function identified on the template will alert the team to a possible recommendation to remedy that lack. A published register of patents and trademarks, an overseas technology observatory linked to the embassies, or an information service for industry with access to international sources are examples of candidates for such recommendations.
The source of more specific recommendations will be the virtual case studies. These will identify not just gross features like absence of an office or service, but weaknesses in laws and inefficiency in services that do exist on the National Knowledge System map. It is also here that the non-knowledge specific recommendations will arise, when participants see obstacles to success of the sentinel enterprises that are part of broader problems. But the greatest value will come from the identification of specific inefficiencies: shortage of entrepreneurs or managers, management that does not recognize its key leadership role in encouraging innovation, regulations that are not published, agencies that are corrupt or understaffed, services that do not serve, universities whose graduates are unprepared, or leadership that is unaware of the importance of knowledge and technology. When these are superimposed on the National Knowledge System map, that data that must be collected and the officials and knowledge users who must be interviewed will become apparent, and a program of data collection and interviews will be carried out to reinforce and quantify the recommendations. Based on all these inputs, the recommendations, in the form of a program of remediation, bridge-building, and implementation, may be designed to stimulate sustainable knowledge-based development in the country.
The recommendations and an implementation plan will be prepared by the knowledge assessment team, supplemented by local experts drawn from the virtual case study teams. The global experts, familiar with benchmarks, and the local experts, familiar with local conditions, will provide the balance of opportunity and realism that characterizes a workable plan.