5
CRITERIA FOR ASSESSMENT

Dr. Susan Berkowitz presented a conceptual approach to the framework for assessing practice that included both the context and criteria for identifying good practices. The context was discussed in Chapter 3. This chapter addresses the criteria for assessment.

Dr. Berkowitz presented an overview of the draft criteria that the committee developed to begin a discussion at the workshop on what criteria would be useful for DOL/ILAB to employ in developing a list of standard practices. Dr. Berkowitz noted that the draft criteria were in no way meant to be a finished product, but were a “starting point for a lot of work and discussion.”

The criteria posed by the committee drew on existing literature. In particular, the committee regarded the ILO’s document Time-Bound Programme: Manual for Action Planning, released in 2001, as a useful place to start thinking about criteria for assessing practices. Dr. Berkowitz noted that the planning committee “looked at the way this document defined good practices ( see Box 5-1) and drew from it, although we did not necessarily accept it wholesale. There was a lot of discussion about which of the criteria that were laid out in this document we thought would be appropriate for our particular uses.” The committee agreed on these criteria specifically because they were general enough to allow for consistent evaluation across a range of practices implemented in varying contexts.

The committee’s draft criteria (Box 5-2) consisted of six concepts: impact, program/practice effectiveness, replicability, sustainability, cost effectiveness, and relevance.1 Dr. Berkowitz discussed each of the six elements in the draft criteria. (In the handout produced for the workshop—and reproduced below—a series of questions was provided to help identity important themes within each criterion. The questions themselves are not the criteria, however.)

Impact was meant to be taken broadly, going beyond the confines of the program or the particular practice. Did the program reduce child or forced labor, assuming that was—or should have been or would be—the larger goal around any such program. Did it benefit child or forced laborers? How was impact measured? Dr. Berkowitz noted that it was important that some sort of evaluation, even if not a formal one, be applied to the practice. She noted that it was important to include both direct and indirect or unintended consequences. She noted that these latter consequences can be quite large.

1

Dr. Berkowitz noted that these terms are not necessarily universal so it is very important that we define what we mean by the terms for the purposes of this workshop because there may be somewhere else were people would not use these terms this way. There is no absolute convention about the meaning of these terms.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 55
5 CRITERIA FOR ASSESSMENT Dr. Susan Berkowitz presented a conceptual approach to the framework for assessing practice that included both the context and criteria for identifying good practices. The context was discussed in Chapter 3. This chapter addresses the criteria for assessment. Dr. Berkowitz presented an overview of the draft criteria that the committee developed to begin a discussion at the workshop on what criteria would be useful for DOL/ILAB to employ in developing a list of standard practices. Dr. Berkowitz noted that the draft criteria were in no way meant to be a finished product, but were a “starting point for a lot of work and discussion.” The criteria posed by the committee drew on existing literature. In particular, the committee regarded the ILO’s document Time-Bound Programme: Manual for Action Planning, released in 2001, as a useful place to start thinking about criteria for assessing practices. Dr. Berkowitz noted that the planning committee “looked at the way this document defined good practices ( see Box 5-1) and drew from it, although we did not necessarily accept it wholesale. There was a lot of discussion about which of the criteria that were laid out in this document we thought would be appropriate for our particular uses.” The committee agreed on these criteria specifically because they were general enough to allow for consistent evaluation across a range of practices implemented in varying contexts. The committee’s draft criteria (Box 5-2) consisted of six concepts: impact, program/practice effectiveness, replicability, sustainability, cost effectiveness, and relevance.1 Dr. Berkowitz discussed each of the six elements in the draft criteria. (In the handout produced for the workshop—and reproduced below—a series of questions was provided to help identity important themes within each criterion. The questions themselves are not the criteria, however.) Impact was meant to be taken broadly, going beyond the confines of the program or the particular practice. Did the program reduce child or forced labor, assuming that was—or should have been or would be—the larger goal around any such program. Did it benefit child or forced laborers? How was impact measured? Dr. Berkowitz noted that it was important that some sort of evaluation, even if not a formal one, be applied to the practice. She noted that it was important to include both direct and indirect or unintended consequences. She noted that these latter consequences can be quite large. 1 Dr. Berkowitz noted that these terms are not necessarily universal so it is very important that we define what we mean by the terms for the purposes of this workshop because there may be somewhere else were people would not use these terms this way. There is no absolute convention about the meaning of these terms. 55

OCR for page 55
REDUCING THE USE OF FORCED OR CHILD LABOR 56 BOX 5-1 ILO Good Practices • Innovative or creative What is special about the practice that makes it of potential interest to others? Note that a practice need not be new to fit this criterion. For example, often an approach may have been in use for some time at one setting, but may not be widely known or have been applied elsewhere. • Effectiveness/impact What evidence is there that the practice actually has made a difference? Can the impact of the practice be documented in some way, through a formal programme evaluation or through other means? • Replicability Is this a practice that might have applicability in some way to other situations or settings? Note that a practice does not have to be copied or “cloned” to be useful to others. • Sustainability Is the practice and/or its benefits likely to continue in some way, and to continue being effective, over the medium to long term? This, for example, could involve continuation of a project of activity after its initial funding is expected to expire. But it could also involve the creation of new attitudes, ways of working, mainstreaming of child labor considerations, creation of capacity, etc., that could represent legacies of a particular practice. This criterion may not apply to all types of practices. • Relevance How does the practice contribute, directly or indirectly, to action of some form against child labor? • Responsive and ethical Is the practice consistent with the needs, has it involved a consensus- building approach, is it respectful of the interests and desires of the participants and others, is it consistent with principles of social and professional conduct, and is it in accordance with ILO labor standards and conventions? • Efficiency and implementation Were resources (human, financial, material) used in a way to maximize impact? SOURCE: IPEC, Good Practices: Identification, Review Structuring, Dissemination, and Application. Geneva: ILO, October 2001, pp. 2-3.

OCR for page 55
CRITERIA 57 BOX 5-2 Criteria for Assessing Practices Draft Proposed at Workshop to Facilitate Discussion 1. Impact a. Did the program reduce child or forced labor? b. Did it benefit child or forced laborers? c. How was impact measured? d. Direct, indirect, and unintended consequences? 2. Program and/or practice effectiveness a. Did the program achieve its goals? 3. Replicability a. Could the practice be implemented with modest adaptation in other settings? b. What factors limit/encourage replicability? 4. Sustainability a. Is the practice likely to continue (as needed)? b. Is the benefit likely to continue effectively? i. Is the institutional capacity necessary to sustain benefits or practices? ii. Local ownership 5. Cost effectiveness a. Were the benefits adequate in relation to likely benefits from comparable investments? b. How is cost effectiveness measured? 6. Relevance a. Is there a set of assumptions about how activities will lead to outcomes? (Do you understand why the program works?) b. Is there a logical connection between the inputs, activities, and outcomes? Program or practice effectiveness—the second element—was more focused on the program or intervention and asked whether it achieved its goals. It is a somewhat narrower construction than the notion of impact. Replicability focuses on the extent to which a practice is unique to a particular situation or can be used elsewhere. What kind of modifications would be needed to have a program that is successful in one venue be successful in other venues without significant adaptation? A related question is to ask which factors limit or encourage replicability, because that helps one to understand the context in which a practice can work or not. Sustainability is, in a sense, the other side of replicability and focuses on whether a practice can continue beyond an initial outlay of resources (e.g., initial funding). Individual programs or initiatives are often funded for a set period of time and then financing is reduced or eliminated (sometimes with the idea that other sources of funding will be found by the time the initial period ends). This is very common in almost any

OCR for page 55
REDUCING THE USE OF FORCED OR CHILD LABOR 58 kind of initiative, according to Dr. Berkowitz. Then the question has to be, can it continue on in the absence of this funding? What sort of lasting infrastructure or activities did the initial funding produce? This raises the further question of institutional capacity building: has a program laid the groundwork, for example, via local ownership, so that it can continue? Cost effectiveness was meant to focus on the balance between the infusion of resources and the outcomes of the practice. When the committee was crafting the draft criteria, Dr. Berkowitz suggested that it was concerned about practices that might work very well, but at very high cost. Businesses ought to have a methodology for measuring cost effectiveness and incorporate this concept into the strategic thinking. Relevance focuses on the underlying theory linking the practice to its desired outcomes. Is there a set of assumptions about how activities will lead to outcomes? This came from a discussion about logic models, which are often used in program evaluation. Logic models identify the inputs, the activities, and the outcomes (short term and longer term) that are assumed to derive from that. The connections between those three elements are important. Dr. Berkowitz suggested it is important to understand the connections between these components of practices. One reason is that to judge whether a program has been effective, one needs to understand these assumptions. Mr. Viederman (Verite and a committee member) noted that the committee believed that in order for something to be considered a good or good enough practice not all of these six elements must be present; for example, a program that has a large positive outcome and is cost effective but not replicable might still be a good practice. Comments on the Criteria Participants at the workshop—both presenters and audience members—reacted to the draft criteria in various ways. One presenter said they were useless and another too generic. Others found them very useful. Most offered suggestions for clarifying, adding to, or changing the criteria presented. This section begins with general comments about the criteria, followed by comments made to specific criteria. (Additionally, the National Research Council received written comments on the criteria, which have been reproduced in Appendix G.) General Comments 1. There is no need for the criteria, because they already exist. Bama Athreya from the ILRF suggested that there was no need for new criteria as criteria already existed: “Apparently no mapping was done of the number of excellent sets of criteria that have been developed over a period of decades now to ensure (1) that investors do not have the risk of seeing their investments placed in products or companies that use child or forced labor, (2) that the labor rights community has undertaken to see the broad applicability of core labor standards, and (3) that consumer rights groups have undertaken to ensure that consumers are not purchasing goods made by child or forced labor. I have in my folder here just one set of such criteria developed and they are criteria, let me be very clear about that, developed by a European pension fund and spelling out exactly what sorts of due diligence that fund is undertaking with regard to

OCR for page 55
CRITERIA 59 companies it holds to ensure that those companies are employing best practices to reduce or eliminate child labor. These efforts exist. There was no need for this committee to reinvent the wheel and the fact that they chose to do so rather than mapping and simply borrowing from such best practices is troubling.” An audience participant also expressed that the criteria were generic. “That is, [the criteria] could have been picked out of any handbook on evaluation and said, here it is. The criteria as formulated do not reflect any of the experience or wisdom that is obviously present [at the workshop]. So I do not know how one would move…in the direction of formulating criteria that somehow presupposes some understanding or knowledge of what is being dealt with, but the generic quality of the criteria seems to me really emasculates, vitiates any strength it might have as a tool.” 2. Criteria could be better defined. Jorge Perez-Lopez noted that good definitions were critical: “It is going to be a challenge to define some of these things because there are very many situations that arise and how these concepts are defined is very important.” Rachel Rigby commented that “it would be very helpful for our criteria…to also be accompanied by a set of definitions.” Jeff Morgan concurred, noting that terms must be very clearly defined and understood among key stakeholders. In particular, he asked for clarity on the following terms: • “reduce” child or forced labor • “benefit” child or adult laborer • withdrawal, remove, rescue • trafficking–with or without movement One audience member felt that the difference between “impact” and “effectiveness” was unclear. Another noted, “There are also words that are used interchangeably that are not the same. Impact and consequence are not the same.” According to the participant, “usually impact is something that you expect to happen. The consequence you may or may not expect it to happen, but they are not the same. An impact is directly what happens, a consequence may be something that happens down the road, it is longer term. It depends upon how you are thinking about it and so on. I do not care how you define these words, but I think you that to define them carefully so that everybody understands what you are talking about.” 3. Criteria should be reordered. One audience member noted that the criteria are not in the right order so they do not encourage logical thinking about it, about the project or about the program and so on (see an alternative presentation of the criteria in the comments submitted to the meeting in Appendix G). In a related vein, another participant suggested that the criteria be organized chronologically, that is, some elements, like relevance, could be construed as being applied before the program, while impact happened later, followed by sustainability and replicability. Another also argued for a hierarchy, but based on which elements were most important: “Obviously what we care most about is…was it effective in reducing…child labor or forced labor. Whether it was cost effective or not sort of

OCR for page 55
REDUCING THE USE OF FORCED OR CHILD LABOR 60 presupposes that we have a whole universe…of models that work and that we are going to choose among them whether this one costs less in relation to the outcome or not. As best I know, that is not the case so I think there is a need for some sort of hierarchy here. What is important? I would pick out, personally, did it work and that it is replicated several times here. Maybe can it be replicated?” 4. Greater clarity is needed as to how many criteria a practice has to meet to qualify as a “good” practice. As Dan Viederman noted, there was not an expectation when the draft criteria were created that a practice had to meet all of them to qualify as good. However, this leaves open the question of how many criteria a good practice should meet. Rachel Rigby similarly added: “There are six criteria here. There are some subcriteria and I am sure there is no definitive answer, but exactly how many need to be present in order for it to be a good practice. You mentioned that it did not need to be all. I can think of some programs where they maybe have one, but the others are sort of weak and then I do not know if I would call that a good practice.” Jeff Morgan brought up the question of how practices would be scored or how elements would be weighted. This comment could be seen as addressing how much each criterion is counted among the six proposed, how practices would be scored in meeting each criterion or how observers would score practices. He asked for example, “What determines being placed on the DOL list or removed from the list?” 5. Context needs to be further emphasized. Jeff Morgan argued that the criteria appear to be focused on remediation efforts (programs) only. Mr. Morgan thought the criteria were too program or project focused. He was concerned that the criteria do not consider the other elements that enable the necessary programs. The criteria, in his view, need to look at the enabling environment that would allow projects to take place. As he outlined it in his presentation: • The criteria lack context of the environment in which the labor issue occurs. o What are the key issues faced by those who seek to address the problem? Overall recognition, commitment, and capacity to address the issue o What laws and regulations are in place and enforced in the area where issues are occurring? Legal, regulatory, and judicial framework o What is the stability of the government and the government structures in the region where the issue takes place? Is the government supportive of a plan to address the issue? Mr. Morgan mentioned that the level of recognition and type of involvement on the part of the government with regard to a problem made it more or less difficult to work with and define programs. Mr. Morgan provided the Côte d’Ivoire, the largest producer of cocoa in the world, as an example. He noted that the stability of that government has been in question since 2001 and it is still creating a lot of issues for industry when it comes to putting programs on the ground in the country. Second, he argued that the criteria are narrow in scope and may be based on erroneous assumptions regarding the scope and root causes of the problem. He pointed

OCR for page 55
CRITERIA 61 out differences he perceived between: o “Hired” labor vs. family labor o Agriculture vs. manufacturing o Subsistence vs. commercial agriculture o Economic status of families in the sector o Status of infrastructure: schools, roads, health centers, education, communication capabilities. Mr. Morgan said it was important to distinguish between employment activities, for example, between factories hiring laborers and family labor. He also argued it was important to distinguish between different types of agriculture and between agriculture and manufacturing. He noted that he had met farmers whose children had to work or the family would go hungry. He also noted the issue that the preferred alternative, that is, school, might not be available. Toni Dembski offered that there was a need for understanding the objective of a business entering into one of these programs, that is, the objectives could be broad in a number of respects. She also offered that a contextual element is the scope of the importing that might be undertaken by a company. In the case of Target, its breadth— 3,000-4,000 different contractors and perhaps up to 10,000 different factories—is a complicating factor. An audience participant suggested that the criteria be grounded in three kinds of contexts. First, there is the environment in which the agent of child’s work, the parent or family, makes decisions about a child engaging in work that benefits the child and the family. It is very hard to come in and impose judgment from the outside about whether we should try to defeat that or work with that and so on. Second, there is the legal environment, which may include laws against child or forced labor or some sort of exploitation, which the participant took to be the focus of this meeting. It is important to note that the laws and the ILO Conventions do not bind farmers, they bind governments, and the government may simply lack the regulatory authority force to put the laws into affect. In a third environment the government itself is the culprit. The participant pointed out Turkmenistan and Uzbekistan as primary examples of this. In cases such as these there is a nexus. “Democracy has been mentioned, freedom of association, but invariably child labor in these environments is imbedded in a social and political context where you just cannot go in and solve the problem of child labor. You have journalists in jail or journalists being shot. You have political activists or trade unionists who are in jail or who were suppressed or whose organizations are managed by government. To talk about child labor in this environment without placing it in a larger…political context in which these abuses occur…is really to live in a never- never land. … It seems to me if we are going to talk about child labor in that third context, we really need to talk about and address all of the other problems here.” 6. Other components could be added to the criteria. Bill Guyton urged that the use of partnerships could be a criterion. He argued that there was a need for the commitment of all stakeholders--whether government, private sector, civil society and farmers--in the case of cocoa.

OCR for page 55
REDUCING THE USE OF FORCED OR CHILD LABOR 62 Mr. Guyton also suggested that an incentive structure was a critical element that seemed to be missing from the criteria. A few audience members took up this point, noting that there was an important distinction between positive and negative incentives. One audience member summarized a few points on incentives: “I think you emphasize here regulation, compliance, and monitoring, but it is hard to regulate if the incentives are working against you and it becomes much easier if you look at the incentive structure and try to build your intervention so that some people are doing what they want to do or what is beneficial for them. I think that means first of all finding partners. In the ILO, trade unions often can be brought on board and sometimes even employer associations when they view child labor as an issue that creates an unequal playing field. Finding and using incentives also, I think, involves what I would call piggybacking, that is building your program on top of another program that is working already so you do not have to do everything yourself.” Bama Athreya noted that transparency and traceability were missing: “The criteria that we are missing or rather the subjects that seemed to be missing altogether in the development of the criteria are supply chain transparency and traceability, transparency so that consumers can have access to information about all of the suppliers in a supply chain, traceability so that companies themselves can have a system with which they can track all products back to the point of origin.” Specific comments on individual criteria Impact Mr. Morgan highlighted two issues with regard to assessing impact. First, he noted the goal of the Harkin-Engel Protocol: “We were asked to implement a system that would show that cocoa was being produced without any of the worst forms of child labor.” He noted that if the objective is 100 percent eradication, “it is going to always be difficult to say that yes, that objective is being met.” Mr. Morgan also noted that the Protocol includes deadlines that the industry has struggled to meet and that progress can be subjective. Thea Lee suggested more attention be paid to the notions of direct, indirect, and unintended consequences, “because certainly with respect to child labor, the unintended consequences are an enormous issue for folks evaluating child labor programs. Do you send the children begging or other even worse outcomes, prostitution and so on, if you take them out of the apparel supply chain and so it seemed like in the criteria, you would want very explicitly to ask whether the program design incorporates elements to directly mitigate the unintended consequences and evaluate those elements of the program separately and give it the attention that is deserves.” Bill Guyton spoke to the issue of practices within the context of preexisting programs. He asked, “How do you look at labor criteria fitting into existing programs or preexisting programs that you may have already been operating and how do you begin to measure practical inputs and outputs for those programs?” One audience member spoke about the difference between qualitative assessment and quantitative assessment. “It is one thing to know how many students are enrolled in school. That is not the same thing as how many students are able to attend school. Also, what does it mean for somebody to complete their education? Are there exams? Is there

OCR for page 55
CRITERIA 63 a measure of quality or is it just you enroll and you show up for x percentage of the year and therefore you get passed on to the next grade and so on.” Program and/or practice effectiveness The main comment on this issue was that it was not clear what effectiveness meant. As noted above, defining terms is critical. Replicability One audience member noted that there is a difference between a practice being replicable and a practice being scaled up. Sustainability Jeff Morgan noted with respect to sustainability that the effort may be more than an industry by itself can promise. He argued that sustainability requires capacity building within the country, sustainable sources of funding, and a long-term commitment. Mr. Morgan noted that sustainability is a big question when you talk about a large industry and a large sector. He argued that it depends not only on the industry’s interest, but on the interest of the government as well. Bill Guyton asked, “How do you know over time if a program is truly going to be sustainable if it will start and end and then evaporate?” Cost effectiveness Toni Dembski noted that it was not clear what cost effectiveness means and how it is measured. For example, is the practice cost effective for the business that is operating it or is it cost effective in some other context? One audience member noted: How do you measure it? I get really concerned with the word “cost” because usually the cost we talk about is how many dollars or euros or whatever currency we are talking about could cost. When we talk about cost effectiveness, though, we also have to take a look at the cost to the children if the program is not in place and the cost to the forced laborers if the program is not in place. So when we use the word cost, what is it we are talking about? Are we talking about just the financial cost or are we also talking about the human cost when something like this does not take place?” Relevance Jorge Perez-Lopez questioned whether relevance needed to be a criterion: “I am still struggling with this issue of relevance and whether it is a criterion or whether this is just a factor or a methodological issue. I do not see how relevance really is or should be a criterion in your work.” An audience member noted that it is really of critical importance to get the logic and the logic model explicit and suggested moving that up to the beginning and to recognize that there are many logic models for child labor, that is, there are many reasons why there is child labor. One of the pitfalls, the commenter noted, in the design of child labor programs is a mismatch between what we think the problem is and what the solutions are. The audience member gave three examples to illustrate the point. One was a child labor program in Central Asia that assumed that the problem was poverty and the solution was to buy milk cows for each family in a village, so a good milk cow was the

OCR for page 55
REDUCING THE USE OF FORCED OR CHILD LABOR 64 intervention based on the notion that children were working because the parents were not able to earn adequate income. A second program, in Russia, was aimed at dysfunctional families, providing family counseling for families with alcoholic parents and so forth. The assumption here being that the problem was a family problem, more sort of sociology. Third, in Central Asia too often migration is a cause of child labor. When people migrate to pick cotton or some other product, they take their children with them and there are no schools around. “So it really depends on the logic model and getting that right. I think that is just a critical first step.”