Keeping What You Paid For—Retaining Essential Consultant-Developed Knowledge Within DOTs (2017)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 C H A P T E R 2 2.1 Overview Figure 2 outlines the basic steps covered in this guide that facilitate capturing and learning consultant-developed knowledge. These steps are: 1. Identify needs for capture and learning of essential consultant-developed knowledge—on a project-by-project basis or for different categories of work that the agency outsources; 2. Define specific objectives for knowledge capture and learning that consider types of knowledge, learning objectives for DOT staff, and intended methods for application of knowledge gained; 3. Identify suitable strategies that match with needs and objectives; 4. Select specific knowledge capture and learning techniques for each of the identified strategies. Sections 2.2 through 2.4 cover steps 1–3 above. Step 4 is covered in chapter 3. 2.2 Identify Needs for Knowledge Capture and Learning The first step is to identify which outsourcing engagements should be targeted for knowledge capture and learning activities. This can be approached on a project-by-project basis, or, more strategically, by looking at streams of work or categories of activities. A DOT may choose to implement an active and deliberate approach to capture and learning of consultant-developed knowledge in order to (1) mitigate risk and/or (2) take advantage of opportunities that may improve future practice or build more capable internal staff resources. Risks Depending on the complexity and characteristics of the work being outsourced, insufficient transfer of knowledge from the contractor to agency staff during the engagement may pose risks to the DOT. If the work is routine, well-specified, and accomplished in accordance with the specifications, and staff already possess substantial knowledge of how to perform that type of work, risks may be minimal. On the other hand, risks will be greater if the work was complex and involved any of the following: • Deviations from standard procedures, • Use of innovative methods that are outside of agency staff expertise, • Extensive interaction or negotiation with multiple parties within or external to the agency, or • Unusual circumstances or events. Framework for Capturing and Learning Consultant-Developed Knowledge

6 Keeping What You Paid For—Retaining Essential Consultant-Developed Knowledge Within DOTs Key risks to consider are: • Misinterpretation or misapplication of consultant work products, which can lead to down- stream problems (including liability issues); • Inability for agency staff to answer questions about the assumptions or decision processes inher- ent in the consultant’s work, leading to a loss of agency credibility with customers or partners; • Lack of understanding of informal agreements made by the consultant during the course of the work, which can result in broken commitments and degradation of internal and external relationships, in addition to increased need for re-work; and • Inability to follow through on consultant recommendations, ultimately diminishing the value of the agency’s investment in the consultant’s services. Missed Opportunities Lack of knowledge transfer can also mean lost opportunities for discovery and future applica- tion of lessons learned and effective practices developed. For example, an agency may change vendors or consultants as a result of a contract re-compete. If no knowledge transfer took place from the original vendor to the agency, the new vendor will need to start from scratch, without the benefit of prior experience that the agency has paid for. In addition, without knowledge transfer, agency staff responsible for contract development and oversight cannot apply lessons learned to anticipate specific problem areas in future contracts. Over time, some DOTs have shifted certain agency functions (e.g., design) to being predomi- nantly outsourced. Once an agency stops performing a particular core function, existing exper- tise to support that function erodes and can be challenging to rebuild. Of special concern is that maintaining a strong consultant oversight function when the DOT is no longer performing a particular type of work itself becomes especially difficult because there is no natural “pipeline” of junior engineers moving up the ranks and gaining the experience necessary for overseeing contractor work. Lack of sufficient in-house capabilities for proper contractor oversight is an important area of concern for DOTs given a host of associated safety, liability, and cost escala- tion risks. Effective knowledge transfer from consultants can help to sustain and build employee capabilities, providing a base of experience from which they can draw on for future initiatives. Of course, the viability of knowledge transfer strategies depends on the availability of DOT staff to be on the receiving end of these strategies. Assessing Needs for Consultant Knowledge Capture and Learning Given the discussion of risks and opportunities above, consultant knowledge transfer needs can be classified into three types: • Remembering what was done and why. Retaining site- and project-specific knowledge to inform future maintenance, operations, design, construction, and customer liaison related to that project/location. 1. Identify Needs (section 2.2) 2. Develop Objectives (section 2.3) 3. Identify Strategies (section 2.4) 4. Select Techniques (Chapter 3) Figure 2. Process for planning consultant knowledge transfer activities.

Framework for Capturing and Learning Consultant-Developed Knowledge 7 • Improving future practice based on experience. Avoiding repeat mistakes and identify- ing and replicating effective engineering, design, and construction practices. This type of need is aligned with the continuous improvement process disciplines that are in place in many DOTs. • Developing seasoned engineers and managers. Building program and project management experience including negotiation, contractor oversight, stakeholder liaison, issue identification and resolution, and schedule/scope/budget control. As DOTs lose their cadre of veteran senior managers, outsource more of their work, and shift from a construction focus to an operations and maintenance focus, there is a need to develop new competencies. These competencies can be built by providing opportunities for sustained interaction and collaborative learning involving consultants and internal staff. Each of these reasons will be applicable for many—but certainly not all—situations. For example, a major project involving innovative techniques that represent uncharted territory for the DOT would be a target for knowledge transfer activities; a small scale consultant project for highly routine work that follows a well-established set of procedures would not. And even where knowledge transfer is desirable, there is a need to distinguish what is most important to focus on given limited resources. For any given consulting engagement or stream of outsourced work, agencies can use the tool provided in Table 1 to clarify knowledge capture and learning needs and determine a focus for knowledge transfer activities. The following are steps to use this tool: 1. Identify which criteria are applicable to the consulting engagement. 2. For each applicable criteria, count the number of X’s shown for each of the three types of needs. 3. Consulting engagements with at least one X for a particular column are candidates for deliberate knowledge transfer activities for that particular need. 4. Knowledge transfer activities can be prioritized based on the number of X’s assigned. 2.3 Develop Knowledge Capture and Learning Objectives Once the needs for knowledge capture and learning are identified, specific objectives can be developed for (1) what types of knowledge should be captured, (2) who should be involved in the learning process, (3) what the nature is of the learning that needs to take place, and (4) how the agency can leverage what has been learned. Developing these more specific objec- tives will help to guide selection, design, and implementation of appropriate knowledge transfer techniques. Question 1: What Do We Need to Know? Types of knowledge to be captured and learned will vary based on the underlying type of need. Remembering What Was Done and Why. For this type of need, the emphasis is on project-specific knowledge that may not be documented as part of the standard deliverables. In addition to the basic facts of the project (what, when, where, who, and why), DOT staff will need to be able to defend the design decisions or recommendations that were made when they receive questions from external stakeholders or colleagues in the agency. They will also need to bring an understanding of agreements, key assumptions, and rationales for design choices to future work on the same facility to avoid unintentional contradictions with earlier work or failure to follow through on commitments.

8 Keeping What You Paid For—Retaining Essential Consultant-Developed Knowledge Within DOTs The following may need to be understood in order to field future questions or ensure continu- ity of future actions at the project site: • Who are the people that championed this project—both within and external to the agency? • Who was responsible for technical decisions? Management decisions? • What agreements or commitments were made to external stakeholders as part of this work— both formal and informal? • What were the most important choices that were made during this project, and what was the rationale for making them? • What was done that differed from the original scope/plan? • What precipitated the changes—e.g., external events, new requirements, new information, incorrect assumptions, etc.? Improving Future Practice Based on Experience. For this type of need, the goal is to build on what is learned from earlier engagements to avoid repeat mistakes and improve upon past techniques for greater efficiency or effectiveness. The Criteria for Application Needs Remembering what was done and why Improving future practice based on experience Developing seasoned engineers and managers Large scale project X X X Potential for extensive elapsed time between delivery of consultant products and use (e.g., time lag from design to construction) X Likelihood of inclusion in future audit X Consultant not likely to be available for continuing support or assistance following project completion X Work involves extensive interaction with multiple external stakeholders X X Work involves potentially controversial elements X X Work involves high degree of technical complexity X X X Work involves application of new/innovative features or techniques X X X Fast-track project—highly accelerated timeline X X X DOT identified a need to expand in- house capability to perform this type of work X DOT has limited staff with expertise required to oversee consultants in this area X DOT has a gap in highly experienced managers due to retirements of veteran staff X Table 1. Assessing needs for knowledge transfer.

Framework for Capturing and Learning Consultant-Developed Knowledge 9 focus of knowledge capture is on improving future practices; this involves analyzing what hap- pened on past projects and drawing conclusions about implications for the future. Key questions to consider include the following: • What technical or management challenges were encountered during the engagement? What strategies were used to address these challenges? • If a project like this arises again, what should be done differently? • Were any new methods used that should be replicated in the future? • Were any specific helpful resources (information, people, tools, etc.) used that others should take advantage of in the future? • What are the most important things that teams should know before starting similar projects? Developing Seasoned Program Managers and Project Engineers. This final category of need is the broadest and most variable. Essential knowledge may include understanding of how to perform particular activities or how to navigate challenging situations. Specific learning outcomes should be created, consistent with the roles and associated develop- ment plans for staff involved. These should consider technical and managerial skills as well as softer skills such as negotiation and communication. Question 2: Who Are the Target Learners? An essential element of a knowledge capture and learning strategy is targeting DOT staff who will interact with the consultants before, during, and at the close-out stage of the engagement. It also involves identifying others in the agency who may not have direct contact with the consul- tants but who are expected to utilize or learn from what they produced. Because staff turnover within DOTs occurs, it is important to identify not only individual staff members but also the roles within particular business units that these individuals play. This provides the basis for internal knowledge transfer activities. Remembering What Was Done and Why. For this type of need, questions are: • Who will take ownership of the consultant’s deliverables? • Who will use the consultant’s deliverables—or supervise contractors or other consultants that will use them? Improving Future Practice Based on Experience. For this type of need, questions are: • Who should be involved in the process of identifying lessons learned from a particular project? • Who should understand and “own” the lessons learned from a technical perspective—i.e., be able to explain them to others? • Who will be most likely to apply these lessons in the future? Developing Seasoned Program Managers and Project Engineers. For this final category of need, questions are: • What opportunities are there to build internal DOT staff capabilities through involvement in this project? • Who should be involved? At what levels and in what capacities?

10 Keeping What You Paid For—Retaining Essential Consultant-Developed Knowledge Within DOTs Question 3: What Is the Best Way to Capture and Learn This Knowledge? One common pitfall in planning knowledge transfer activities is to assume that just because knowledge is captured (e.g., the consultants deliver a report with pertinent information or a lesson from a project is recorded in a lessons learned database), it will be learned. The process of learning is complex and multi-dimensional. It is worth spending some time to consider what the learning objectives in a given situation are and how to best achieve them. Table 2 presents a framework for developing learning objectives based on a modified version of Bloom’s Taxonomy.1 This taxonomy has been widely used in education circles for curriculum planning and offers useful insights into more complex thought processes involved in learning that go beyond the ability to memorize and recall facts. In the table, rows are types of knowledge; columns represent cognitive processes. Knowledge types comprise the following: • Factual. Knowledge of basic terminology and of specific details that are needed to understand a topic, event, or situation. • Conceptual. Knowledge of interrelationships among basic elements; classifications, catego- ries, principles, theories, models, and structures. • Procedural. Knowledge of how to do something; understanding of techniques, procedures, methods; knowledge of criteria for when to apply different methods. • Meta-Cognitive. Knowledge about cognition itself—knowing what you know; knowing how to teach others; understanding what level of thinking is required for different tasks. Cognitive processes comprise the following: • Remembering. Recalling relevant knowledge from long-term memory. • Understanding. Constructing meaning from inputs (written, auditory, visual) through inter- pretation, classification, inference, comparison, or summarization. • Applying. Executing or implementing a procedure. • Analyzing. Breaking a body of information down; understanding how the parts relate to the whole, understanding structure and interrelationships. • Evaluating. Making judgments based on criteria and standards. • Creating. Developing or producing something new by reorganizing known elements into a new pattern or structure. Remembering Understanding Applying Analyzing Evalua ng Crea ng Factual Conceptual Procedural Meta- Cogni ve Table 2. A framework for developing learning objectives. 1Anderson, Lorin W., David R. Krathwohl, and Benjamin Samuel Bloom. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Allyn & Bacon: Boston, MA.

Framework for Capturing and Learning Consultant-Developed Knowledge 11 Knowledge and cognitive processes become increasingly complex and abstract as one moves from top to bottom and from left to right. Tables 3–5 illustrate differences in typical areas of emphasis for learning objectives for three types of consultant knowledge capture and learning needs that have been identified. The shading of cells in the tables indicates the required knowledge types and cognitive processes for achieving certain learning objectives. Table 3 represents the first type of need: “Remembering what was done and why.” A specific learning objective here could be to make sure a staff engineer is able to recall which version of a standard was used for a particular project. For this area, the primary requirement is recall and comprehension of facts, concepts, and procedures. Achieving this learning objective can be accomplished quickly through a simple verbal set of instructions or through direction to appropriate written materials. Table 4 represents the second type of need: “Improving future practice based on experi- ence.” This requires individuals to analyze and evaluate past activities and results, understand cause-and-effect relationships, and create and apply new procedures. Achieving this learning objective requires focused activities and collaboration. Table 5 represents the third type of need: “Developing seasoned program managers and project engineers.” Here the focus is on building procedural and meta-cognitive knowledge, Remembering Understanding Applying Analyzing Evaluating Creating Factual Conceptual Procedural Meta- Cognitive Table 3. Learning objectives: remembering what was done and why. Remembering Understanding Applying Analyzing Evaluating Creating Factual Conceptual Procedural Meta- Cognitive Table 4. Learning objectives: improving future practice based on experience. Remembering Understanding Applying Analyzing Evaluating Creating Factual Conceptual Procedural Meta- Cognitive Table 5. Learning objectives: developing seasoned program managers and project engineers.

12 Keeping What You Paid For—Retaining Essential Consultant-Developed Knowledge Within DOTs since the objective is to build seasoned managers who will be able to hire and develop staff and modify processes as needed. To achieve this learning objective, an extended period of experiential learning with mentoring will be needed. Question 4: How Can the Agency Leverage This Knowledge? The final question involves thinking beyond the learning process to how the knowledge will actually be applied to benefit the agency. Questions to consider for each of the three need types are listed below. Remembering What Was Done and Why • If a question comes up on the project after the consultant completes their work, what process will be in place to direct the inquiry to the right unit or person? • What processes will be used to ensure that agreements and commitments made during the project will be kept in the future? • What processes will be used to ensure that future work at the project site fully considers the basis for prior decisions? Improving Future Practice Based on Experience • Who will be responsible for incorporating what was learned (if appropriate) into modifications to agency manuals, policies, procedures, or guidance documents? • How will the DOT ensure that relevant lessons are understood and used by DOT staff and other consulting teams embarking on similar projects? Developing Seasoned Program Managers and Project Engineers • What learning outcomes are being sought? • Who will be responsible for tracking progress and evaluating whether desired outcomes are being met? Considering each of these questions will provide some clarity on what the agency is trying to achieve, and what elements will be required to: • Capture essential consultant-developed knowledge, • Ensure that learning strategies are in place that match the nature and complexity of the type of knowledge to be transferred, and • Ensure that organizational strategies are in place to obtain value from application of this knowledge. 2.4 Identify Knowledge Capture and Learning Strategies The next step is to identify knowledge capture and learning strategies that match the established objectives. At a high level, there are three types of strategies for knowledge capture and learning: • Documentation and Handoff Processes. Developing consultant work scopes, deliver- able formats, and end-of-project handoff processes that incorporate knowledge transfer requirements. • Identifying and Learning Lessons. A combination of (1) targeted activities in which partici- pants in a project or stream of work are asked to identify what has been learned that could be of value to others and (2) activities to ensure that these lessons will be applied—for example, cov- ering them at lunch-and-learn sessions, building them into before-action reviews, including them in training curricula, or considering them in updates to standard operating procedures or

Framework for Capturing and Learning Consultant-Developed Knowledge 13 guidelines. It is important to note that both types of activities must be implemented in consort for this strategy to be effective. An over-emphasis on capture without sufficient attention to learning will not achieve the desired end result, which is an improvement to practice. • Mentoring and Interaction. Using blended consultant–DOT staff teams, utilizing consultants to provide one-on-one mentoring to DOT staff, and other techniques that provide extensive person-to-person interaction in an experiential setting over an extended time period. A framework for matching the needs that have been identified for knowledge capture and learning to the three strategy types is shown in Table 6. The shading in the table shows the primary strategies for each purpose/need for knowledge capture and learning. By design, each of the three strategy types corresponds to one of the three types of needs: • If the purpose is to ensure that knowledge about what was done on the project and why is prop- erly captured and understood, documentation and handoff processes are essential, primary strategies. • If the purpose is to improve future practice based on experience, the primary strategy is to identify and learn lessons. • If the purpose is to develop seasoned program managers and project engineers, the primary strategy involves establishing mentoring and interaction processes involving the consultants and the target staff. In reality, DOTs will have a mixture of needs and will use multiple strategies in combination. For example, a brainstorming session at the end of a project to discuss lessons learned can sup- port all three of the purposes: it can help DOT staff review what happened in the project and ask questions (remembering what was done and why) as well as identify lessons learned (improve future practice based on experience). In addition, participating in reflective discussions about projects can build staff capabilities as they can draw upon these lessons as they face similar situ- ations in the future (develop seasoned engineers). Knowledge transfer can be achieved through several different mechanisms: via delivery of documentation (including codification of lessons learned and effective practices), through formal training, and through active learning involving person-to-person interaction between agency staff and the contractor staff. Chapter 3 draws on a series of case studies to provide further information on knowledge transfer techniques. Purpose/Need for Knowledge Capture and Learning Strategies Documentation and Handoff Processes Identifying and Learning Lessons Mentoring and Interaction Remembering What Was Done and Why Improving Future Practice Based on Experience Developing Seasoned Program Managers and Project Engineers Table 6. Matching needs to strategies.