Introduction

The National Academies of Sciences, Engineering, and Medicine was tasked by the Veterans Health Administration (VHA) to prepare a comprehensive resource planning and staffing methodology guidebook for VHA Facility Management (Engineering) Programs. The resource and staffing methodology must take into account all significant parameters and variables involved in the VHA Engineering Programs. The methodology should yield customized outputs based on site-specific input data, to enable specification of the optimal budget and staffing levels for each site (see Appendix A for a full statement of task).

Topics and considerations to achieve this task include the following:

• How VHA Engineering performs operations and maintenance now.
• Details of a mix of knowledge, skills, and abilities (KSAs) required to perform operations and maintenance.
• Details of the complexity and diversity of VHA facilities.
• How staffing is currently performed.
• How modeling works for operations and maintenance.
• How the situation is expected to change in the future.

Currently, the VHA does not utilize a staffing model for defining its facilities workforce. Each medical center defines its required facilities staffing. This interim report focuses on the types, availability, usage, and limitations of models in the staffing processes. Because this is an interim report, recommendations provided in this report are limited to model data.

The Importance of Staffing Models for the VHA

The Federal Property Management Reform Act of 2016¹ requires federal agencies, with respect to the use of federal real property, to perform workforce projections to assess the property-related requirements of the federal workforce. Apart from this obvious mandate, a staffing model enables quantitative planning of VHA Engineering staffing targets, which in turn provides the foundation for human resources activities. Qualitative aspects of workforce planning are dealt with through other processes. Such a model can be used at various levels of aggregation—specifically at the overall (national) VHA level, at the Veterans Integrated Services Network (VISN) level, and at the site director level. At the national level, use of models support exposition of data used in presentations made to Congress for changes in staffing levels (increase or decrease) based on the known requirements for building/system inventory, required service levels, and current staffing, which in turn can be used to prepare and justify VHA budget requests. Also, at the national level, such a model provides a tool for VHA compliance engineers to determine whether adequate staff is

___________________

¹ Public Law No. 114-318, Federal Property Management Reform Act of 2016, H.R. 6451, 114th Congress, December 16, 2016.

available to comply with U.S. Department of Veterans Affairs (VA) directives and program requirements. At the VISN level, a model can be used to equitably allocate staffing resources between sites, as these same parameters change at the site level. At the site level, the director would have quantitative access to where facilities engineering demands are likely to exceed current staffing in coming years so that funding resources can be allocated optimally between the various functions within a VHA site. Any staffing model should be able to guide leadership at all three levels.

As an example of the potential benefits of a comprehensive staffing model, consider the current practices observed in how individual Veterans Administration Medical Center (VAMC) directors appear to manage their funding budget, keeping some in reserve for dealing with unforeseen events or emergencies.² Unforeseen events are a reality for management, but a staffing model could reduce the guessing associated with managing an unknowable future. Any excess in the unforeseen events/emergencies fund can be allocated to a list of shovel-ready projects at the end of the fiscal year, but is this optimal? Directors have many demands on their limited budgets beyond facilities engineering but have staffing models only in some specific professional areas such as nursing or occupational safety and health. Knowing the demands for continued safe operations and tolerable risk levels, directors can make more informed decisions about budget allocation and reserve/emergency amounts. Perhaps more importantly, a robust staffing and funding allocation model would allow the VISN directors to better manage and allocate funding toward prioritized investments rather than leaving funding in such a dispersed manner. It would also allow appropriate levels of staffing and funding at all locations. This would be a more valid procedure than the current reliance on past staffing and “feel,” as one director put it to the committee. Naturally, these uses of a facilities engineering staffing model demand that a valid and reliable model be developed and implemented. Any such model must also recognize that staffing cannot be changed instantly. Staffing levels have inherently more inertia than budget dollars because of the time lag required for necessary staff changes, especially in recruiting and hiring replacements.

VA Facilities—Size and Complexity Issues at VISNs and Sites

The VHA includes both headquarters and regional sites spread across the United States and select overseas locations. Each site is intended to serve a geographical location with specific healthcare needs. Unlike many enterprises, the VHA building inventory has sites of different ages, and often there is a mix of building size and age at each site or campus. The buildings themselves contain a variety of functions with extremely different space and infrastructure demands. The complexity includes such factors as operating rooms, research laboratories, outpatient facilities, spinal cord injuries centers, domiciliary or other long-term skilled nursing facilities, and orthopedic treatment areas. They also have differing needs for utilities and contain different equipment, both building services, and biomedical systems. Specifically, the VHA has 172 sites, grouped into 18 VISNs, organized regionally. Each site has a chief engineer (a degreed position) with a staff of engineers, tradespeople, and administrators to ensure that the facilities remain operable and meet all requirements for medical staff and patients. Sites differ in the number of staff in Facilities Engineering, even for sites that appear comparable in size, complexity, and age.

Differences in sites may result from a variety of challenges, such as unique regional needs or remoteness. Some sites may have needs, such as water treatment facilities or fire departments that are not universally required. Other sites may be in relatively remote areas (e.g., rural areas) where resources are few, both in terms of total staff and of the availability of outside resources such as contract workers. One staff illness in such sites is far more difficult to cope with when there is only a single plumber or designer

___________________

² Currently, maintenance is funded by each director using funds allocated from the number of patients entering the facility. The director can hold back funds for their individual facility for an emergency. So too can the regional leadership hold funds in case something happens to one of their many facilities. Of course, a manager will retain some funds for emergency use, but if annual emergencies do not exhaust the funds, then they can be allocated, presumably for a prioritized list of ready-to-go projects.

than when there are many, as at a larger, urban site. Also, with the changing patient population, some buildings may not be needed but must still be maintained at some level for safety reasons. With sites typically comprising one or more self-contained campuses, it is not easy to find alternative uses for or to sell underutilized buildings.

COMPLIANCE STANDARDS

The VHA complies with the requirements of the Joint Commission^3,4 as well as the standards. As presented by Steven Broskey and David Alvarez at the committee’s first meeting, there are requirements labeled by the VHA as “above and beyond” covering specific operations and facilities.⁵ The term “above and beyond” is used by the VHA to denote issues beyond the usual compliance standards (e.g., Joint Commission) applicable to all medical facilities. The term does not imply that these issues have only recently been acknowledged by the VHA: Some have a long history of being addressed in depth by VHA Engineering. These comprise the following VHA directives:

• Boiler Plant Operations
• Legionella
• Sterile Processing Services
• Safety and Health During Construction
• Electrical Power Distribution Systems
• Physical Security Design Manuals
• Women’s Health Program

These are a sample of many other directives and federal laws that apply to VHA Engineering. All of these are complied with using established procedures and internal audits. External audits include those by the Office of the Inspector General, Joint Commission, Occupational Safety and Health Administration (OSHA), Environmental Protection Agency (EPA), Nuclear Regulatory Commission (NRC), Veterans Administration Central Office, and VISNs.⁶

The Scope of Facilities Engineering Activities

There are two broad ways to derive staffing models: extrapolating from the status quo or by preparing a task-based staff structure. An extrapolation is done based on current staffing using the variance between sites to fit statistical models to the staff data. A task-based staff structure is prepared by listing all of the tasks required of staff and how long each takes, thus building up estimates of the total workload. This section covers task lists for the three main functions of Facilities Engineering: operations and maintenance (O&M), capital projects, and engineering administration. Two additional functions (capital asset data management, project management, and project delivery) are included where appropriate. Note that typical VHA engineering departments consist of engineering administration, project management, maintenance

___________________

³ Mills, G. 2018. The Joint Commission. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, September 26.

⁴ The Joint Commission is an independent nonprofit organization that accredits and certifies more than 21,000 health care organizations and programs in the United States.

⁵ Broskey, S., and Alvarez, D. 2018. VHA Engineering Resourcing and Staffing Study Sponsor Presentation for National Academies Committee. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, September 26, Slide 98.

⁶ Broskey, S. 2018. VHA Engineering Resourcing and Staffing Study Sponsor Presentation for National Academies Committee. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, September 26.

(skilled trades), grounds maintenance, operation of high-pressure steam, operation of chiller plants, biomedical engineering, and safety (including overall medical center safety, fire safety, emergency management, industrial hygiene, and EPA compliance). In some facilities, there may be additional maintenance activities required for the research laboratory and medical equipment. Maintenance can also include regular sustainment and restoration projects—such as air handling replacements, an overhaul of patient areas, or comprehensive relamping—where planning and oversight may be different than larger capital projects. Since staffing models already exist for the safety program and biomedical engineering, these are not included here except as they impact the other functions such as engineering administration.

TASKS TO DRIVE THE STAFFING MODELS OF VHA ENGINEERING

Operations and Maintenance

O&M comprises much of the visible daily work of Facilities Engineering. This includes inspecting; planning and executing required repairs or replacements by the trades (carpentry, electrical, plumbing, locksmith, painting); and scheduled maintenance such as preventive maintenance (PM) activities. O&M does not include capital projects or oversight of projects by Facilities Engineering staff. Operations include daily requirements to ensure the provision of all building systems (e.g. conveyance, building automation system (BAS); medical gasses; HVAC, heating/ventilation/cooling, including central chiller plants; steam; etc.), building envelope, utilities (power, including generation in some instances; water and plumbing; etc.) as well as grounds maintenance and janitorial services.

For Facilities Engineering O&M, work assignments are typically controlled by work orders. These are assigned to individual staff members for execution. They may be initiated by either non-engineering staff to cover problems seen in the buildings or by engineering staff noticing such problems on their regular rounds. There are also work orders generated by routine maintenance, such as PM of scheduled work on equipment. Last, there are special projects initiated by management for such activities as installing new equipment or even building new facilities or modifying existing buildings. Some of these activities can be undertaken by means of contracts (see the next section). It should be noted that for each VAMC, Joint Commission accreditation requirements drive significant workload for the O&M section because they must meet 100 percent preventive maintenance of all equipment per manufacturer requirements and perform extensive testing, inspection, and maintenance of all building components and systems.

In theory, analysis of the times taken to complete work orders can be used to determine workload. At a more detailed level, the individual tasks within each work order can be listed and analyzed. For example, time studies or similar industrial engineering techniques can provide direct time estimates of tasks independent of work order analysis. Chief engineers, who typically have spent time working at lower organizational levels, may be one source of time estimates, or a set of subject-matter experts (SMEs) can be assembled and tasked with using their experience to derive time estimates. It is crucial to also account for travel time on top of the actual work order requirements. Many VAMCs operate numerous buildings or are responsible for other sites/locations that require travel. Note that all task time estimates are subject to variability, although this is often ignored in simple time studies. For any tasks that vary in completion time, it is important to capture the variability (e.g., by finding the time distribution) so that more valid estimates of staffing can be obtained to use in staffing models of these inherently probabilistic tasks. Generating models from time studies can be a costly undertaking to perform correctly, but it does represent an alternative to other modeling methods.

Other organizations have developed standard times for tasks for their activities across sites, so this could be applied to VHA Facilities Engineering where similarities exist. Helpful software is available for organizing such data by building and across sites.

Capital Projects, Project Management, and Project Delivery

Larger capital projects in Facilities Engineering are handled as defined projects rather than as part of O&M. The work may be undertaken by in-house project management staff, or through contracted staff, or by a combination of both, depending on specific projects and workload. The larger projects are typically contracted out (as described in this section), but they still require considerable work on the part of Facilities Engineering staff to support the contractors’ activities. The project must be designed and scoped and sent to contracting to solicit proposals, the contracts vetted and approved, the contract awardee worker training organized and completed, and the contract fulfillment supervised and approved for completion. If needed, staff may have to intervene, stop contracted work, require rework, and reinspect after interventions. Also, most of the construction projects take place in a fully operational medical center. This requires extensive planning and coordination to eliminate fire risk, infection control issues, special removal of debris, the noise of construction, delivery of construction material, working hours, utility interruptions, contract workers, security risk, and many other factors not part of typical commercial construction. Because projects are diverse in scope, required skills, and cost, the amount of engineering time required is also highly variable.

Medical facilities submit all minor construction projects (projects that add >1,000 SF) regardless of cost and Non-Recurring Maintenance (NRM) projects above $25,000 for approval through VA’s Strategic Capital Investment Planning (SCIP) process. Projects above $20 million are considered major construction projects. VHA-approved Minor Construction projects and NRM projects above $1 million are submitted to a national SCIP panel, which reviews and ranks budget year SCIP projects for inclusion in VHA’s long-range capital action plan. This process is used to prioritize capital planning projects between and within sites to ensure that capital is used in the most appropriate manner. This obviously requires planning and staff time to use the SCIP database and prepare (potentially competitive) cases for projects. In 2004, the NRC report Investment in Federal Facilities⁷ recommended that federal facilities investment decisions be related to overall mission. SCIP’s strategic goal is to improve the delivery of services and benefits to veterans,⁸ which follows the over NRC recommendation. It is doubtful whether all recommendations of the 2004 NRC report are followed for any specific project, but SCIP appears to be a well-conceived, bottom-up rollout of investment needs, with the overall level of administrative activity depending on density, age of facilities, and geographic location, as well as other factors. While the SCIP process has advantages and provides recognized value, the committee did hear numerous concerns that submitted projects were vetted and approved by VISN leadership with funding allocated in a manner inconsistent with the prioritization of projects and needs within individual VAMCs.

The SCIP process provides money only for capital projects. Each VAMC must provide Engineering oversight to manage the planning, oversight, and final closeout of all projects. Particular functions include detailed design (which may itself be outsourced on occasion), training of contract personnel (with safety training to internal VHA standards, a major consideration), and supervision of contract progress and completion. An important consideration of any capital project is what skills, knowledge, and abilities will be required to operate and maintain the new facility after project completion. Variability in funding for capital projects makes predicting the necessary project staffing difficult and should be considered in resource staffing decisions.

The actual form of the contract is important because different contract vehicles drive the level of administrative input. Contract vehicles include indefinite delivery/indefinite quantity (IDIQ) forms as well as specific contracts for single projects. Other organizations have standardized contract detail across their systems to reduce the time required for the contracting process. There is an opportunity to explore other contracting methods to improve VA project management.

___________________

⁷ National Research Council (NRC). 2004. Investments in Federal Facilities: Asset Management Strategies for the 21st Century. Washington, DC: The National Academies Press. https://doi.org/10.17226/11012.

⁸ Veterans Affairs. 2011. Strategic Capital Investment Planning Process Directive 0011. August 8. http://www.va.gov/vapubs/viewPublication.asp?Pub_ID=575&FType=2.

Engineering Administration

In addition to actually carrying out the O&M work required for safe and efficient patient care, the process must be managed and administered. Part of the administrative function consists of contracting. Contracting, as noted above, is used for larger capital projects and for augmentation of O&M resources requiring specialized skills. An example of the latter would be elevator maintenance, typically a function requiring specific skills and knowledge. At least some part of the administrative function (e.g., chief engineer, core office staff) is perhaps best modeled as a fixed level of staffing, as it is required irrespective of the size of the site. Beyond this, such functions as contract oversight, work order processing, and travel time requirements will drive overall workload, which is often related to VAMC size, care complexity, and patient load. Administrative tools such as Computerized Maintenance Management Systems (CMMS) and Computer Aided Facility Management (CAFM) require sizable resources and staff time to manage and use.

VA chief engineers provided a list of common administrative tasks. Tasks include report writing and presentation, oversight of forms/paperwork required by site directors, design functions required before any new project as well as for ongoing change (e.g., provision of more ergonomic workplaces for staff), preparation for and supervision of external audits, human resource functions, safety planning, and meetings. These are all administrative functions, but require both staff time and chief engineer time for oversight. This task list will form part of the basis for staffing models of administration, particularly as they were from sites of quite different sizes and geographic locations.

Staffing Models—Dimensions of the Task and of Potential Solutions

Arrayed against this diversity of sites and tasks are a number of techniques for building staffing and cost models. The aim is to choose from the available modeling technologies those that are most suitable to match the diversity inherent within the VHA and its staffing needs. At its core, a staffing model requires an estimate of the amount of work to be completed and the necessary time and effort to complete the designated work. This estimate can be inferred from predictor variables, such as the currently used square feet of buildings, the medical infrastructure within the building (i.e., medical gases, air flow requirements, pressurization standards, etc.), time spent on work orders, demands for staffing defined by specified missions or mandated time (i.e., a 24/7 staffing requirement for fire or ambulance services, boiler plants, etc.), and on input from SMEs to find overall estimates of staffing required. However, different predictors may be needed for each element of staff—for example, administrative staff versus specific trades. Matching the modeling approach to the specific needs of VHA is the main task of this committee. Even within a grouping such as maintenance, there can be considerable variation between different crafts. For example, electrician staffing can be quite different from HVAC staff, and the two trades are not interchangeable.

Many agencies employ standardized staffing models for various functions of their organization. Implementation of a staffing model for VHA Engineering functions would be useful in its own right. The VHA appears to rely on a bottom-up approach for creating staffing plans at each site, modified in light of budgetary requirements, both the allocation to a site and the further allocation within each site. The VHA does have considerable data available in resources such as the annual Capital Resource Survey (CAPRES) owned by the Office of Capital Asset Management, Engineering, and Support (OCAMES) or the portfolio of variables available in a database maintained by the VA Office of Productivity Efficiency and Staffing (OPES) to help inform any staffing and resource plan or model.⁹ They also have measures of performance, Joint Commission outcomes, plus internal measures such as the fraction of PM actions completed on time, or the results of satisfaction surveys of staff and patients.

___________________

⁹ Moran, E. 2019. A Staffing Model Approach: VHA Administrative Staffing Mode. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, March 12.

Quantitative staffing models are not new. At least three reports by the National Academies of Sciences, Engineering, and Medicine have considered and recommended models for the Federal Aviation Administration (FAA) Air Traffic Controllers, Aviation Safety Inspectors and Systems Specialist.^10,11,12 These all had to take into account systems diversity, differing levels of expertise among the staff modeled and multiple measures of system performance, making them potentially relevant as prototypes for a VHA Engineering staffing model.

The committee heard from military manpower planning experts (McCulloch, U.S. Army Manpower Planning Analysis Agency; Timothy Clary, U.S. Air Force Manpower Agency); facilities managers at other federal agencies (Scott Robinson, NASA; Daniel Wheeland, National Institutes of Health); public nonprofit academic healthcare system (William Seed, Jackson Health System); private sector healthcare organizations (Don Orndoff, Kaiser Permanente); and private sector property management organizations (JLL; Grant Thornton; Gordion).¹³ There are also models for the U.S. Air Force (Brian Norman) and for higher education (APPA and Brian Yolitz), both of which apply to maintenance systems with their inherent mix of deterministic and stochastic demands for staff. This combination of deterministic and stochastic characteristics implies that at any given level of staffing you can have both user- and over-staffing at different times. As noted in an earlier staffing report for the FAA systems specialists, the balance between the costs of under- or over-staffing and the costs of system outages needs to be evaluated on a risk basis.¹⁴ In addition, several systems within the VA have staffing models developed or currently being developed: these comprise Interior Design Staffing, Nursing, Occupational Safety and Health, Logistics, and OPES.^15,16,17 Thus, there is no shortage of models linked to Facilities Engineering that reflect either or both the nature of the function (e.g., FAA Systems Specialists) and the VHA environment (Interior Design, etc.). There are also more general classifications of staffing models—that is, those based on a statistical analysis of current staffing levels and those based on the summation of task frequency and duration.

To choose between the types of model available, there are a number of considerations. Who will use the resulting model? Is the model intended for use at the VHA headquarters for overall equitability and mission success? Is the model intended for VISN leadership to better ensure the prioritization of facility resources within a geographic area? Is the model planned for use at the individual VAMC level to make sure that directors will emphasize the allocation of staffing within the Engineering section? Or, is the model intended to influence behaviors at all levels?

As noted, the same model can be used at different levels of VHA, and indeed VA, leadership, but first all users must trust the model predictions. Second, how does the level of risk affect the model choice? Risk, which is an expression of the probability of the occurrence and impact of potential problems, is inherent in a maintenance staffing model, because of natural uncertainties in the underlying system. Knowing how given staffing levels will affect the risk to the broader system—that is, patient outcomes and staff safety, is important whether the level of risk is chosen a priori across the entire system or is treated as a variable to be chosen anew for different situations. The other element of risk is budgetary, as both over- and

___________________

¹⁰ NRC. 2014. The Federal Aviation Administration’s Approach for Determining Future Air Traffic Controller Staffing Needs, Washington, D.C.: The National Academies Press, https://doi.org/10.17226/18824.

¹¹ NRC. 2007. Staffing Standards for Aviation Safety Inspector. Washington, D.C.: The National Academies Press, https://doi.org/10.17226/11742.

¹² NRC. 2013. Assessment of Staffing Needs of Systems Specialists in Aviation. Washington, D.C.: The National Academies Press, https://doi.org/10.17226/18357.

¹³ See Appendix B for a full list of presenters.

¹⁴ NRC. 2013. Assessment of Staffing Needs of Systems Specialists in Aviation. Washington, D.C.: The National Academies Press, https://doi.org/10.17226/18357.

¹⁵ Emanuelson, C. and Fletcher, E. 2019. VHA Interior Design Staffing Analysis Tool. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, March 12.

¹⁶ Taylor, B. 2019. Staffing Methodology for VHA Nursing. presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, March 12.

¹⁷ E. Moran, E. 2019. A Staffing Model Approach: VHA Administrative Staffing Model. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, March 12.

understaffing may occur at different times within parts of the organization. In general, the higher the staffing level, the lower the system risk to participants but the higher the cost to the organization. Last, there are considerations of accuracy, reliability, validity, and level of aggregation of the model. Accuracy is the precision with which model outputs are given—for example, to the nearest five electricians or the nearest two electricians. Reliability is a measure of the model’s consistency of output for similar input conditions. Validity is how well the model predictions reflect the reality on the ground: Can the tasks be performed with the level of staffing predicted? Level of aggregation, as already defined, is the level of detail: Do the predictions apply to a specific trade function, to a whole medical center, or only to the overall VHA?

A second classification is by the level of aggregation, as mentioned earlier. Does the committee use the 18 VISNs, or the 172 medical centers, or the quoted total of 5,639 buildings, or even the individual systems within each building? Does the committee treat engineering staffing as a total dollar amount, total number of full-time employees (FTEs), or even numbers of individual trades and administration? Under level of aggregation is also the time period covered by the model: Should it be quarterly, annual, or even every 5 years? This logic applies to all measures, not just the input measures used as examples. The higher the level of aggregation, the less specificity, so that a VISN model using dollars of the budget would potentially give less precision (e.g., explain less of the staffing variance) than a less-aggregated model, but would be much simpler to collect data for and to run.

MEASURES OF FACILITIES ENGINEERING PERFORMANCE

In a function where required tasks and their durations are not always predictable, it is important to understand relevant performance measures so that outcomes for different staffing levels can be predicted with accuracy. For example, in the FAA Systems Specialists report¹⁸, the key performance measure was the fraction of time that the National Airspace System was working correctly, with 100 percent as the goal. To start, the committee needs to further develop how the various measures of engineering performance can be classified, beyond the classification by accuracy, reliability, validity, and level of aggregation noted above.

Many more measures are needed for a comprehensive picture of engineering effectiveness. Obvious examples are the patient outcome measures detailed below, but also patients’ reported experiences and satisfaction. Such data are already collected—for example, measures such as the fraction of time that a patient or member of medical staff is unable to be serviced because of engineering system nonavailability. These are not combined with the existing facilities and staffing data such as CAPRES, which is completed annually by chief engineers and maintained by OCAMES. Even gross measures such as counts of adverse events can be part of the set of outcome measures used to validate a predictive staffing model. As the level of aggregation increases to the VISN or VHA level, such outcome measures take on more salience—for example, with the ultimate funding source of the U.S. Congress. It should be noted that medical centers are one of the most constantly reviewed organizations. For VA Engineering, reviews are conducted by accreditation groups such as the Joint Commission and others. The VA Central Office conducts routine engineering compliance reviews for engineering operations and project management. VA VISN offices conduct annual reviews for safety, EPA compliance, energy, and project management. The VA Inspector General reviews medical centers, including engineering, for various compliance issues. OSHA and EPA both federal and state, occasionally review engineering. One-time spot inspections happen frequently. All these reviews produce a large volume of reports that show the overall effectiveness of the engineering program.

___________________

¹⁸ NRC. 2013. Assessment of Staffing Needs of Systems Specialists in Aviation. Washington, D.C.: The National Academies Press. https://doi.org/10.17226/18357.

In modeling, a classification of major importance is of the type of measure. In many enterprises, including medicine, a classification into input, process, and outcome measures is used:^19,20,21

• Input measures: These tell what enterprise inputs are available to accomplish the mission: number of medical centers, size of each, facilities/systems at each, status of facilities (e.g., age), facilities geographical location, the time and distance requirements, round-the-clock missions that are present at some—although not all—VAMCs, load on the facility mission (numbers of patients, clinical needs, severity, etc.) and of course staff numbers and availability, and finally budget available.
• Process measures: Also known as intermediate measures or internal measures, these describe the internal workings of the enterprise. They are often more easily captured than outcome measures. Examples include use of overtime, use of contracts, the fraction of time that a service or facility is available for use, contracting officer representative (COR) duty demands, control of medical variables such as secondary infections, waiting time for patients, the fraction of admitted patients for whom a bed is available.
• Outcome measures: These are the ones that really matter to the mission. At the extreme end of the process are measures of clinical success for patients—for example, impacts on patient access, mortality and morbidity, preventable readmissions, and patient satisfaction with their VAMC experiences.

A perfect staffing model would relate the staff required directly to outcomes that matter to patients and to those funding the facilities—that is, Congress. Realistic models seek to relate staffing to lesser measures of outcome, or even to process measures. The challenge is to balance the costs of data collection and modeling with the strength of model predictions. A model based on process measures would be simpler and less costly than one based on, for example, patient outcomes but may not have the same effectiveness to model users at different levels of aggregation—for example, when justifying Facilities Engineering budgets to Congress.

Inputs—Data Requirements for Models

A perfect staffing model would strike a balance between the specificity of predictions and cost/time of feeding the model with data. The balance chosen would depend on the user of the model: U.S. Congress for determining overall budgets, down to a chief engineer at one medical center wrestling with staffing needs during peak vacation times. The data requirements for any model are a key consideration to address before deciding on the model structure. Inputs may be different for each function or subfunction. For example, project management will have factors based on project workload and other variables, while the electric shop will be based on various compliance requirements and a number of user-generated work orders.

Any database for use by a staffing model needs to include the elements pertinent to staffing levels. This means that it needs to go beyond just details of buildings at each medical facility, although these details are of course fundamental. For example, measures of performance relevant to Facilities Engineering are needed so that staffing models can be related to outcomes as well as to facilities inputs. The current annual CAPRES

___________________

¹⁹ Drury, C.G. 1984. Emergency Medical Services: Human and System Reliability. Le Travail Humain, v47, No. 1, pp. 75-87. http://www.jstpr.org/stable/40657260.

²⁰ Gibson, G. 1974. Guidelines for Research and Evaluation of Emergency Medical Services. Health Services Reports, Vol.89, March-April 1974, pp. 99-111.

²¹ Gibson, G. 1976. EMS Evaluation: Criteria for Standards and Research Designs. Health Services Reports, Vol. 11, Summer, pp. 105-111.

TABLE 1 Suggested Data Sources

contains building and site information and also current staffing levels and contract details and is one possible approach to acquiring some of the data necessary for modeling. An initial listing of the necessary data can be compiled from the data provided in the workshops and meetings and has been classified under headings that will potentially suggest more data sources (see Table 1).

The committee has learned from VHA presentations that much of the data in this table already exists to a varying degree of accuracy but is not integrated into a single trusted data set. The choice of platform and format of this single data set is an important step toward developing accurate, reliable, and valid staffing models and therefore requires considerable thought. How to merge data from several existing databases, how to collect currently unrecorded data, and how to ensure timely and accurate updates are all important considerations.

However, the committee has heard consistent comments about a general lack of accuracy in the input of base-level data into existing engineering data systems, and this is important to discuss. Even if a new data management system were identified and selected for implementation, if there are existing and serious concerns about the accuracy of data entered by harried staff at the local level who are not experiencing any negative consequences to poor data input, is there confidence that a new data system will produce better aggregate information? As part of any potential solution, it would be important for the VHA to consider what the training program would be, how the data would be used, and what the consequences are for good, or poor, data entry.

As noted earlier, a seamless system is needed for data entry, so that accurate data can be assured to base models and funding decisions on. This means that those performing data collection and data entry are a major determinant in ensuring that any model provides valid results. An obvious assertion is that existing validated data should be the first priority for entry into the model’s database. If data exist and have already been entered for other purposes, then one source of potential human error in data entry is avoided. Also, from human-system integration studies going back to the 1970s, it is known that the people doing the data

entry need to see some benefit to themselves, not just to those who want the data.²² When the data have been entered, there should be a means of checking the data for logical inconsistencies, such as zero staff or funding for a particular group of staff—for example, electricians, who may have been misclassified as chief engineer office staff.

More specifically, for Facilities Engineering there is a database of considerable magnitude available, showing mainly input measures: the Capital Resource Survey (CAPRES). CAPRES shows the data at each site (medical center) with over 80 variables, including the expected expenditure measures (total, research, contracts, leases), complexity index, various measures of site acreage, numerous square foot measures (owned, leased, used, etc.), building ages and replacement values, and current staffing at various levels of aggregation (total, trades, administration). CAPRES was developed for use by chief engineers, although more people, in fact, use it currently. The data within CAPRES that was provided to the committee is not wholly reliable, perhaps due to those inputting the data not totally understanding its classification.²³ For example, some sites show zero staffing or zero expenditures in categories where nonzero values would be expected. At higher levels of aggregation (VISN, VHA) some chief engineers see CAPRES as useful if “outliers” are omitted, although that is not good statistical practice.²⁴ At least the CAPRES data allows some statistical analysis, although, from a staffing models perspective, it shows what is rather than what should be. As an example, some initial review of the data by the committee showed that a large fraction of the variance in overall engineering staffing can be predicted from only two variables. Similar results, using three variables, were obtained by VHA staff reinforcing that modeling is potentially feasible.²⁵ Some areas of CAPRES were more reliable than others. The Office of the Chief data showed considerable unexplained variation, and there was an indication that those inputting the data did not understand which staff were to be included or excluded from this function. However, skilled crafts data and project management data were reliable, especially when used with other national benchmarks. The grounds staff are perhaps accurate enough for analysis, but the database did not distinguish between open areas and areas more intensely landscaped. Before any more serious analysis can be performed, the data in CAPRES will need to be validated—not an inexpensive process. The data collection system—for example, training of input staff—does need to be made more consistent.

One major issue is the merging of disparate data sets in a logical and consistent manner. CAPRES could potentially be a basis for this but would need other data on relevant measures. Other VA databases may also include unique but relevant information that is not currently included in CAPRES, such as information in the database maintained by the VA Office of Productivity Efficiency and Staffing. In developing a staffing model, one method is examining the intercorrelations of existing data. For an adequate model, this implies that potentially a large number of variables be available for model building, far more than the final model is expected to use. Merging of data sets requires care and expertise, as they will have been originally devised for different purposes. Personnel with expertise in each database need to work together so that those developing models truly understand each data set and its limitations. As an example, CAPRES does not contain patient outcomes data or patient/staff ratings of quality of care or environment, although these are potential variables that would be helpful in assessing and predicting staffing adequacy. Neither does CAPRES cover some process measures, such as audit report outcomes, adverse events or even completion rates of planned maintenance. It has been suggested by VHA staff in Meeting 3 (see Appendix B) that an index of quality of care or quality of engineering performance could be a useful measure in assessing the

___________________

²² Eason, K. 1988. Information Technology and Organizational Change, Bristol, Pa.: Taylor & Francis/Hemisphere. https://doi.org/10.1201/9781482275469.

²³ CAPRES data provided to the committee.

²⁴ Discussion of chief engineers at Small Workshop 3. Or additional information see National Academies of Sciences, Engineering, and Medicine. 2019. Facilities Staffing Requirements for the Veterans Health Administration EngineeringAdministration: Proceedings of a Workshop in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/25450.

²⁵ Ibid.

adequacy of staffing.²⁶ Also suggested was an index of infrastructure complexity to complement the existing one of clinical complexity for each site.

Finding: Accurate data regarding the functional and physical characteristics of VA facilities is required in order to implement a staffing model for Facilities Engineering.

Finding: Any database used for a staffing model needs also to include staffing and outcome data relevant to Facilities Engineering performance.

Recommendation: The Veterans Health Administration should ensure that the choice of platform and format of facilities database is given careful consideration to ensure continual completeness, accuracy, reliability, and validity of the data.

Recommendation: Policies, procedures, and training for entering and updating information for any comprehensive database, such as Capital Resource Survey (CAPRES), and work order data should be implemented by the Veterans Health Administration to support staffing and other management models.

Overall, the workshops and meetings have shown a growing trend in integrating data systems in the facilities world, to include an increasing number of tools that support integration. Where integration has been successful, there has been a reduced burden on the facility workforce for data entry.

The dimensions along which sites vary are key to model development. Site size, traditionally measured by square feet of used building space, is clearly an important first consideration, but alone it cannot capture all of the information necessary to predict staffing. Clinical complexity, as measured by the VHA’s Facility Complexity Level Model, reflects the types of procedures carried out at a site, the patient volume, and several other factors that appear to affect staffing levels. Age of building and systems within them are other relevant considerations, as are considerations for mandated use of VHA facilities in response to local and regional emergencies. The ability to outsource activities by contract vehicle is also important. The topography and land use of the whole site—for example, woodland versus lawns versus flower beds—will clearly affect the number of grounds staff required. The presence of special activities such as a fire department and the specific requirements for continuous staffing of boiler rooms (mandated in some jurisdictions) are also staffing-dependent. Last, there is the condition of the buildings, reflected in the FCI, and the parallel measure of the quality of care for grounds.

Clearly, a staffing model must take into account some subset of these variables to be logically and statistically valid, but the issue is one of how many are actually needed. For example, some of these variables are correlated with each other—for example, square feet and number of patient visits. In practical terms, the issue may be one of how few variables are needed to build a model of sufficient reliability and validity for use by administrators at all levels of aggregation.

Challenges of Change Management When Implementing Models

When implementing a model, there are always change management challenges. Some specifics are discussed below. Staffing models and the final level of staffing should be based on valid science showing how long tasks take for completion, not on the funding available in the current cycle. This issue will need resolution.

Any staffing model should have face validity as well as statistical validity for it to be well-accepted by users. This requires a balance between using perhaps obscure predictors and believability to those who must

___________________

²⁶ Discussion with participants at Meeting 3 of the Committee on Facilities Staffing Requirements for Veterans Health Administration.

rely on the model. If users cannot see some rationale for each predictor variable, they will have an excuse to stop using the model predictions in their work. An obvious example would be a predictor variable having a negative regression coefficient even though that variable is expected to increase workload. This issue is one of transparency and should not be overlooked in the modeling effort. In multiple regression models, there may a number of predictor variables that may themselves be correlated. This can lead to the statistical anomaly of the coefficient of one predictor being negative because part of its contribution was already covered by a more powerful predictor.

More care will be needed in data gathering and input to feed the model if the model is to be sound. This requires a deep understanding of how each data source is classified, and not placing too much reliance on existing data sources merely because they happen to exist and would, therefore, be less expensive than collecting data anew. New data will be required each year (at least) so personnel will need to be trained carefully to enter data correctly. A quality assurance plan will be needed to ensure the long-term validity of data and hence of the staffing model.

CHALLENGES AFFECTING THE FUTURE ROLE OF VHA

Although the committee was not specifically tasked with examining the effect of ongoing and future changes in the mission and environment of the VHA, modelers, and users of models, do need to be aware of how these changes will affect staffing. Previous NRC reports have discussed how future events and drivers will affect a model.²⁷ From the standpoint of facility maintenance trends in medical services, policy changes, as well as the location of the veterans, become drivers for the model. The trends in medical services include the following:²⁸

• The veteran population is aging and subject to an increasing burden of (usually) multiple chronic diseases, frailty, musculoskeletal disorders, and associated limitations in mobility, vision, hearing and other functional capacities.
• Clinical care capacities are also labile, and new diagnostic and therapeutic technologies imply new interventions and modified outcomes.
• Care that formerly required inpatient facilities is now to an increasing degree provided in outpatient settings.

An obvious current change in policy is the VA Choice initiative, through which individual patients now have some choice (Veterans Access, Choice, and Accountability Act of 2014) in where they can receive treatment.²⁹ To quote: “Veterans who are unable to schedule an appointment within 30 days of their preferred date or the clinically appropriate date, or on the basis of their place of residence, to elect to receive care from eligible non-VA health care entities or providers.” This could have a potential effect on the modeling of staffing at VHA facilities. Note that the VA Choice Act also provides a basis for costing the adverse effects—for example, a VHA facility being temporarily unavailable—as the cost of going elsewhere because of an adverse event is now known. According to VA official documents, roughly 9 million of the current 20 million veterans use VHA services.³⁰ As the predicted veteran population steadily declines from 20 million down to 12 million over the next 30 years, coupled with the fact that less than half

___________________

²⁷ NRC. 2013. Assessment of Staffing Needs of Systems Specialists in Aviation. Washington, D.C.: The National Academies Press, https://doi.org/10.17226/18357.

²⁸ Broskey, S., and Alvarez, D. 2018. VHA Engineering Resourcing and Staffing Study Sponsor Presentation for National Academies Committee. Presentation for the Committee on Facilities Staffing Requirements for Veterans Health Administration Meeting, September 26, slide 98.

²⁹ Public Law No. 113-146, H.R. 3230, Veterans Access, Choice, and Accountability Act of 2014, 113th Congress, August 7, 2014.

³⁰ Veterans Health Administration. 2019. https://www.va.gov/health/ [05/21/2019].

are even eligible for VA medical benefits along with the impacts of the VA Choice Act, the committee assumes that there will be a continued effort to reduce the VHA facilities footprint.

Another trend discussed in various data-gathering meetings and workshops the committee held is for health care to be provided in an outpatient or short-stay setting.^31,32 This allows patients to spend less time in residence, reducing both costs and the potential exposure to secondary infections. Obviously, staffing for both clinical and facility engineering are potentially affected.

The demographic shift of veterans’ residences away from traditional areas of the Northeast and Upper Midwest to areas farther South and West is another consideration for the modeler. In aggregate, the VA population is steadily declining. Almost all areas of the nation are losing eligible populations of veterans, some areas very dramatically. A few are holding steady or gaining. The implication for engineering staffing is that facilities will have lower utilization levels in the areas losing veteran population and higher utilization in the rare areas gaining that population.³³ However, existing facilities are not easily scalable, particularly downward, as buildings may be difficult to repurpose or sell to non-VHA organizations, as noted earlier. Again, there is a potential impact on modeling and on budgeting, as models derived for current demographic distribution and existing buildings may not include the set of predictor variables appropriate in the future. A recent Government Accountability Office report, Improvements in Facility Planning Needed to Ensure VA Meets Changes in Veterans’ Needs and Expectations, explains in detail the planning needed for the future.³⁴ As the VHA designs a model, it should be able to take into account future events and drivers.

IMPROVING SYSTEM PERFORMANCE IN ENGINEERING

Throughout the meetings and workshops, the committee has heard from the management and engineering functions of other organizations that there are many techniques available for improving the organization and performance of Facilities Engineering. Again, improving performance was not part of the committee’s statement of task, but performance improvements will affect the staffing levels required, similar to the external changes noted above. Such techniques include, but are not limited to, benchmarking³⁵ (already in use to some extent by engineering), continuous improvement techniques, quality improvement techniques such as Six-sigma and Lean, better design and enforcement of written procedures, increased consistency in use of the work order system, and software better fitted to its human users.

__________________

___________________

³¹ National Academies of Sciences, Engineering, and Medicine. 2019. Facilities Staffing Requirements for the Veterans Health Administration Resourcing, Workforce Modeling, and Staffing: Proceedings of a Workshop. Washington, DC: The National Academies Press. https://doi.org/10.17226/25456.

³² Government Accountability Office (GAO). 2018. Actions Needed to Improve Oversight of Community-Based Outpatient Clinics. April. GAO-18-375. https://www.gao.gov/products/GAO-18-375.

³³ GAO. 2019. Improvements in Facility Planning Needed to Ensure VA Meets Changes in Veterans’ Needs and Expectations. June. GAO-19-440. https://www.gao.gov/assets/700/699723.pdf.

³⁴ Ibid.

³⁵ Coffelt, D., and Hendrickson, C. 2017. Fundamentals of Infrastructure Management. Pittsburgh, Pa. https://doi.org/10.1184/R1/5334379.v1.