The workshop’s third panel included five speakers who were asked to present on types of fastener failures, root cause analysis methods, key findings in subsea fastener failure analyses, failure reporting, gaps in understanding of root causes, research and development needs to better identify and mitigate root causes of failures, assessment of end-of-life, feedback mechanisms to improve design practices, and factors for successful fastener performance.
Trent Fleece, BP
Trent Fleece is the blowout preventer (BOP) operations team lead for BP’s subsea BOP reliability team working within the global wells organization. The subsea BOP reliability team is engaging in new ways of working with drilling contractors and BOP suppliers to improve the reliability of subsea BOP equipment. Fleece has worked for BP 18 years in a variety of roles within the global wells organization. As team lead for BOP Operations at BP, he is at the front lines of the issue of subsea BOP bolt failures. A BOP’s mission is to stop any blowouts from becoming catastrophic, and Fleece’s job is to deal with any issues that impair the ability of the BOP to shut in a well.
In his presentation, he gave an overview of BOP equipment, reviewed several failure analyses, explained an innovative collaboration to address failures, and detailed his group’s comprehensive action plan for bolts.
Fleece noted that he gathered his source material from the publicly available information on BSEE’s website. He also clarified that he uses the term “bolt” to refer to the fasteners or connectors that are the focus of this workshop; BOPs also have two components known as “connectors” that are not bolts. He also pointed out that BP regularly uses EAC as a blanket term for all environmentally assisted cracking instances, including hydrogen embrittlement.
Fleece began by sharing images of BOPs and describing their components (Figure 4.1). The lower marine riser package (LMRP) is the top half of the BOP and houses two annulars and the control pods. The BOP is controlled via electrical currents from the surface to those pods, which activate the mechanisms by which the BOP functions. The control pods are “the brains” of the BOP. In the case of a hurricane, the well can be shut in and the LMRP and the riser can be pulled to the surface to move the rig to safety.
Below the LMRP are the shear rams. There can be up to seven rams in a typical subsea BOP, two of which are shear rams; the shear rams demark the area of criticality on the BOP system. If a bolt fails above the shear rams, the shear rams shear the drill pipe or casing to create a seal to mitigate a spill. A failure in a bolt below the shear rams makes it very difficult to contain a well. Fleece pointed out that there is a lot of variety between BOPs. Each shear ram has several different types of bolts on it, and BOPs are designed differently by each of the three major manufacturers.
In the Gulf of Mexico, ambient conditions at 6,000 feet below sea level are roughly 2,700 psi of external pressure, and from 3,000 to 10,000 feet, the temperature range is between 34 to 37 degrees Fahrenheit. While laboratory testing is meant to duplicate these conditions, Fleece noted that laboratory test results rarely match in-service performance, a discrepancy he suggested could be reduced by using hyperbaric chamber testing at seafloor temperatures.
API Standard 53 covers BOP equipment and requires predeployment BOP pressure testing to the working pressure. Typically, the LMRP is pressure tested to 10,000 psi and 15,000 psi on the lower BOP rams. A rigorous pressure test at 15,000 psi ensures bolt integrity. Once a BOP is in use, the pressure it sees during deployment is often far lower than the surface test pressure. This heavy load at predeployment testing could be why BP has seen failures at the surface but not at depth, Fleece suggested.
Learning from Failures
Fleece shared details from six bolt failures in the Gulf of Mexico, from different operators, between 2003 and 2015. There were failures before 2003, but today
there are far more rigs operating than in the 1980s and 1990s; Fleece posited that the increased failure rate is likely a function of this increased activity, not drilling depth. In fact, most failures occur around 5,000 feet water depth, because that is where most deepwater rigs are active, he said.
BSEE identified these failures and issued a safety alert to Gulf of Mexico operators in 2016. The operators and BSEE formed a working group to address the safety alert and create an action plan to reduce failures. The group examined the variables in each case and noticed that several failures had a few factors in common, including hardness, coating, and failure mechanism (Figure 4.2).
As Herman Amaya had pointed out in his presentation, failure is a universal throughout the industry. The failures the group studied were from five different operators, four different drilling contractors, and all three original equipment manufacturers (OEMs). Fleece also noted that although these events are categorized as “failures,” that term refers only to the mechanical integrity of the bolt. These failures did not actually result in loss of well control.
In these six failures, the same component made of the same material failed six different ways, although the failures do share some characteristics. Fleece said his interpretation of this is that the human factor plays an important role in root cause analysis. In fact, understanding and improving the human factors aspect was far harder than the technical issues the work group explored. There is not a history of or established mechanism for sharing root cause analysis data within the industry, and there are commercial and legal hurdles to overcome.
Fleece also pointed out that although BOPs and their rams have rigorous bolting standards following the requirements in API 16A, there had been only basic requirements for bolts installed on a BOP. He noted that people in multiple industries tend to underestimate the importance of bolts, often calling them “dumb iron.”
Some of the questions the work group formulated to address failures included the following: What was the failure mechanism (such as HE, fatigue, ductile overload)? Where did it occur? Were there intergranular fractures, and what were their loads? BOP bolts can be quite large, weighing up to 50 pounds in some cases, and they are engineered to be fairly robust, he said.
Fleece described the four root cause analyses that he participated in as follows:
- Riser bolt insert failure (2003). In this case, it was the insert that failed, not the bolt. It was an intergranular fracture, and the failure occurred about halfway down the riser, so the riser failed but the well was secured by the BOP.
- API flange bolt failure (2014). Eight 3-inch studs failed at the first engaged thread, which is where they are inserted into the connector. One bolt failed at the nut and was discovered when a worker put his hand on it. The failure occurred after being in use for 6 months. They had also, just days before failure, passed a pressure test, signaling that they retained their pressure integrity. These bolts were coated in zinc, and what the team noticed was that the zinc had been almost completely consumed after 5 months of service. The zinc coating was designed for surface corrosion, but was in subsea use.
- LMRP connector body bolt failure (2014). In this case, the upper half of the BOP connector became separated from the bottom half. Microtesting during root-cause analysis revealed banding with a 10-point hardness differential in the material. BSL3 dictates ingot casting to prevent this, and the investigation revealed that a sub-vendor used continuous casting. That was just one element of the failure, but an important one, Fleece said.
- API flange bolt failure (2014). These bolts had a “mixed-mode” failure, meaning that there were several issues, including HE, intergranular failure, and ductile overload. The bolts “unzipped,” meaning that a crack formed in the middle of the bolt then spread outward.
Fleece said BP has thoroughly examined environmental loads since its riser break, using a remotely operated vehicle (ROV) to monitor the water current and accelerometers to measure displacement of the riser and BOP, which indicates how much the equipment is swaying. BOPs are designed for massive environmental load but are actually used at only a fraction of that. Bending can happen if the rig moves off location, but in this case, it isn’t the BOP that fails, Fleece said.
There has been a coordinated, industry-wide response to these bolting failures. The API multi-segment task group, for example, involved a variety of rig contractors, manufacturers, and operators (as Tim Haeberle described in his remarks). Its report, issued last year, had 20 recommendations and emphasized a holistic approach to bolting failures, including determining contributing factors, examining current mitigation strategies, and recommending changes to standards to build a more robust system.
The API bolting work group, in contrast, had a chief focus on short- and long-term changes for existing equipment. Standards are generally for new manufacturing, and can vary in how they address existing equipment, so this group is addressing an important aspect of operations today. This new direction is consistent with the other group’s progress, Fleece said.
The bolting group has defined “critical bolting” as bolts whose failures would cause “loss of containment of wellbore fluid into the environment” and voluntary adopted 20E/F for these bolts. It is also updating critical bolts with a hardness over 35 HRC, improving quality control for third-party manufacturers, adding sampling requirements, updating torquing procedures, increasing overall engineer rigor and oversight, increasing failure reporting, and complying with 20E, which eliminates zinc electroplating and coating.
These are voluntary actions, and the group submits a quarterly progress report to BSEE as part of its self-monitoring. This process, Fleece said, reflects the industry’s realization that waiting for new standards to apply to new parts is not an option; it is imperative to take action now.
Failure reporting is especially critical, and so a joint industry project of the International Association of Drilling Contractors and the International Association of Oil and Gas Producers is studying comprehensive BOP performance, beyond just bolts. A database houses information on BOP component defects from 29 global companies. Under BSEE’s guidance, the database will meet the Code of Federal Regulations failure reporting requirements and is intended to be a catalyst for eliminating these defects and comply with Standard 53’s reporting requirements. Over time, the database is expected to generate statistically significant data that can be shared with manufacturers to improve BOP maintenance, management, and integrity. Sharing data can eliminate the existing knowledge gap, in which OEMs are rarely given feedback on their equipment.
In summary, Fleece expressed his belief that the industry’s root cause analysis process is robust. Four root cause analyses for EAC failures reached similar conclusions, including the need to reduce hardness and understand stress values. Sharing failure reports, and making the companies involved anonymous to avoid imparting blame, will improve the process even more, he said, expressing his belief that the
industry is already rising to these challenges. For example, two separate blind shear rams recently failed at the same time but were investigated together by a team of operators, contractors, and an OEM. The investigators identified a root cause of failure and quickly implemented improvements. In the past, these investigations would have been handled separately by each player, increasing time and costs and reducing information sharing.
The Action Plan
A focus on finding the right material is understandable, but Fleece cautioned that “exotic materials may fail in exotic ways.” Right now, he said, 4340 is the best material that meets 20E specifications and also addresses the issues that Fleece’s group discovered from their failure analyses. There are not many 4340 alloy steel API 20E manufactured bolts in use yet, but none have failed, and their numbers, especially in BOP equipment, are growing year by year.
A comprehensive bolt action plan also addresses design, material, and manufacturing processes, including quality assurance and control for 20E bolts, Fleece said. Eliminating zinc plating and torquing properly are also important, and in-service monitoring can also have a big impact. Every bolt should be inspected on a 5-year plan and replaced at the end of its life span. Any cracks should be reported as “near misses” and thoroughly studied to mitigate larger failures.
Fleece called these practices a “drastic change” for the industry and said coordinated industry actions are working to eliminate subsea bolting failures by promoting research. In particular, his team would like to see testing in hyperbaric chambers to better simulate real conditions. He also recommended improving applicable standards, creating both short- and long-term goals to enhance current equipment performance, promoting failure reporting, and sharing root cause analyses.
Of particular importance is improving communication among companies and with BSEE, Fleece said. Admitting failures or defects has been “uncomfortable” for many in the industry, but in the end everyone came together around the goal of saving lives and protecting the environment by eliminating defects on critical equipment.
Describing BP’s 2003 riser bolt insert failure in the context of the material-stress-environment Venn diagram, Fleece noted that the company reduced the hardness and reduced the stress on the bolt by making the bolt longer and thicker. Although they didn’t change the environmental load or CP scheme, “moving” two of the circles seems to have been enough, and there have been no subsequent failures over the past 15 years on multiple rigs using the same type of BOP/riser equipment.
Fleece wrapped up his talk with a note on torquing. 20E includes many torquing specifications. The in-service pressure tests and environmental loads on bolts
are far lower than their installed preload. Torquing is designed to maintain integrity in a wide window of operational conditions and has a wide range of pressures. This flexibility is helpful, but BP is committed to improving its knowledge of what the best torque value really is.
Frank Adamek, Adamek Engineering and Technology Solutions, LLC
Frank Adamek, president of Adamek Engineering and Technology Solutions, has worked in the oil and gas industry for more than 40 years, including as executive chief consulting engineer at GE Oil & Gas. He is an expert in technical interfaces, failure analysis in drilling and production, and technical standards, and is on the board of directors for ASME International (formerly the American Society for Mechanical Engineers).
Adamek has a long history with fasteners and standards. He has been working with API since the 1980s, including a role as chair of the subgroup that developed the flanges standards 6AF and 6AF2. That subgroup examined the pressure, bending, and tension capabilities for flanges to understand how they would behave in the field. He worked with ASME on Appendix Z, the standard for calculating clamp connections next to flanges, and with NASA to develop connecting joints for space shuttles after the Challenger disaster in 1986.
Adamek covered opportunities to improve the reliability of bolts and discussed the need for a high-level assessment of environmental factors that can lead to EAC, with a particular emphasis on the effects of CP.
Steps to Improve Reliability of Bolts
Adamek opened with a brief review of the path forward recommended by the API workgroup and BSEE’s QCFIT report. In particular, “critical bolting,” defined as any connector whose failure could release fluids into the environment, demands special attention. As other workshop presenters noted, the industry is committed to mitigating EAC by reducing material hardness, improving quality throughout the supply chain, and enforcing 20E’s specifications for critical bolt manufacturing. Environmental factors that can generate hydrogen are also being addressed to some extent; for example, zinc has been identified as a creator of hydrogen and a catalyst for hydrogen penetration in bolts, and therefore eliminated from use.
There are additional improvements that could be made to build on this previous work, Adamek said. For example, 20E is an “orphan specification” because it isn’t recommended by other API specifications; recommending 20E in 16A would help ensure proper bolt manufacturing processes. In addition, Adamek discussed
the important role of stress, underscoring the need to control torquing so that equipment is properly calibrated and the bolts have the right amount of torque. The torques these bolts operate at is quite high—30,000 to 40,000 foot-pounds—so it’s important to preload the bolts and ensure static loading in service.
CP and Other Environmental Factors
Adamek said that in general, there has been excellent progress toward addressing EAC through measures affecting materials and stress. Manufacturing and processing procedures have been refined, hardness has been reduced, and bolts are properly stressed. It is the third circle in the Venn diagram—the environment—that must be addressed next, he said. Thus far, steps to identify other environmental elements that are generating hydrogen, such as stray voltages or ground faults, have been “woefully inadequate” in his view. Adamek expressed his belief that the industry must closely examine all operational issues and their potential effects on the equipment, which has also been recommended in the QCFIT reports. Specifically, he pointed to a need to better monitor CP; excessive CP is known to be capable of causing damage, but today CP is monitored only on an ad hoc basis.
Adamek posited that increased bolting failures might be linked to changes in the operation of oil and gas drilling equipment. Roughly 20 years ago, new drilling ships were deployed that used CP systems of impressed currents to apply active voltage to the hull, replacing older equipment that used anodes to protect the hull from corrosion. Just a few years later, in 2002, the bolting failures began.
Proper CP on a drilling vessel is approximately −950 mV, in order to protect the hull without adding undue voltage on connecting equipment. Observations have shown that at this level, the current can travel 3,000 feet down the riser toward the BOP. If the CP is set higher than −950 mV, perhaps in an effort to provide “overprotection” to the hull, it travels down even farther, to 6,000 feet. Clearly, CP is influencing the riser and the BOP, even if it is nominally used only to protect the hull, Adamek said.
At the BOP, there are 40 to 60 anodes made of aluminum and zinc that put out a charge of roughly −1050 mV. Testing with ROVs has shown that the anodes provide a consistent power flow to a BOP’s uncovered structures, and a separate BOP-based CP system protects any scratched paint or damaged coatings. The criteria for proper CP to protect the seal are laid out in EN 12473, which requires regular monitoring of CP levels with impressed current systems (ICSs) to ensure the current is properly controlled. Regular monitoring is not required for galvanic anode systems.
In Adamek’s view, there is a direct, if anecdotal, connection between the failures starting in 2002 and the use of CP on newer ships, which have variable voltage rates with impressed current, as opposed to fixed voltage rates with anodes. All of those
broken bolts were on failures with ICSs; none had galvanic anodes. One failure occurred on a ship with eight ICSs, some of which were out of order. A technician increased the voltage on the working ICS, which likely caused the failure, he said.
Bolting yields in the industry range from 125 to 175 ksi, and the current CP “sweet spot” in deep water is around −850 mV. However, there is a difference between conditions in what we would typically call “deep water” and conditions at 10,000 feet below sea level, and it is not yet known how that increased depth would affect performance, as ROVs have measured voltages as low as −1,500 mV.
Pointing to the fact that cracking occurs even in low-strength bolts, Adamek stressed that addressing material hardness alone will not solve this problem. In his view, controlling environmental factors is “the elephant in the room.” A monitoring system or stray-voltage alarm could alert operators to problems in the equipment; Adamek suggested that if we better understood CP measurements, we could compare them to known deleterious effects, such as cracking at a hardness lower than 34 HRC.
To underscore this point, Adamek reviewed several bolt failures. A 2002 failure that Fleece described was determined to be caused by EAC, but the source of the hydrogen was never found. To prevent repeat failures, industry partners reduced recommended hardness, examined materials, removed coatings, kept impressed current at −950 mV, regularly inspected bolts for cracks, and replaced any that needed them. They also examined causes of EAC and sources of hydrogen and potential EAC damage.
In 2012, LMRP connector bolts failed, and bolt material stress and hardness were reduced, although engineers were careful not to reduce Bolting capacity. Cracked bolts were sometimes replaced with larger bolts, which meant that careful machine adjustments were made to maintain bending, tension, and pressure capability. It took over a year to analyze and replace the connectors, and a safety alert again set the maximum CP charge at −950 mV. The connector failure had bolt heads that were exposed to seawater and could present an entry point for stray voltage, but whether that was a contributing factor was never investigated.
The pattern in both of these instances, Adamek posited, is that the industry addressed in each case nearly everything except the effects of CP. To prevent EAC, the industry has improved the material, improved torquing, and eliminated zinc, but the problem still isn’t solved. We need to look elsewhere, he said. Pointing to data suggesting that at −1250 mV, 34 HRC bolts will fail, he said the time is ripe to further explore the effects of voltage as an environmental contributor to EAC.
Ken Armagost, Anadarko
Ken Armagost is a drilling engineering manager at Anadarko Petroleum Corporation. His team is responsible for providing technical support for drilling engineering (including metallurgical and welding engineering) to Anadarko’s worldwide drilling operations. He provided an overview of root cause analyses (RCA), applied the practice to critical bolts, and suggested more ways the industry could improve reliability.
There are many possible root causes of a bolt failure, including the heat-treating process, threading method, casting process, component design, installation, hardness, and many other factors. The fact that there are so many possibilities creates a need and opportunities for collaboration among scientists, industry workers, and failure specialists. Echoing a few other workshop presenters, Armagost emphasized that working on solutions will require a wider sharing of knowledge among key players in the field.
Overview of Root Cause Analysis
The objective of an RCA is to identify the most basic reason a component failed. While there can be more than one root cause and several contributing factors, essentially a “root cause” is a factor that, if removed from the equation, would allow the system to operate as intended, avoiding any failures.
Many industries and government agencies use RCAs. There are several different RCA techniques, and for these complex critical bolting failures, it might be necessary to combine multiple methodologies, Armagost said. RCAs cover the entire life cycle of the failed component to determine whether the solution affects the design, manufacturing, installation, operation, or maintenance of a component. At the end of an RCA, he said, the result should be actionable. Ideally, an RCA is able to demonstrate that controlling the root cause will provide a solution that can be used to improve performance and reliability.
RCAs for Critical Bolt Failures
Armagost offered some statistics to put bolt failures in perspective. There are as many as 844 critical bolts, studs, and nuts on one deepwater, 7-ram BOP stack. A flanged drilling riser has 900 critical bolts. That is roughly 1,700 bolts per drilling rig, and there are currently 39 deepwater rigs in use, which means there are somewhere around 60,000 critical bolts in the Gulf of Mexico today. Assuming 40 deepwater rigs, continuously operating in the Gulf of Mexico since 2003, each
operating in a water depth of 6,000 feet with flanged risers and using 6-ram BOPs, this amounts to an estimated 800,000 bolt-years of exposure to failure.
Given this context, the fact that there have been relatively few bolt failures means that these are, by and large, very reliable bolts, Armagost said. The new standards addressing BSL2 and BSL3 bolts may further improve reliability. This is the “existing reliability” that RCA is intended to improve upon.
RCA procedures follow a general pattern (Figure 4.3). First, there is an undesired occurrence. Then, a team collects evidence (data and/or physical evidence from the field), selects a methodology, identifies a root cause, develops a solution, implements it in the field, and monitors the performance of the changed component or procedure.
There are several different RCA methodologies used by the oil and gas industry:
- Five “Whys.” In this method, the investigator keeps asking why an event (and all of its precipitating events) happened until he or she arrives at the controllable piece that caused the failure.
- Fishbone/Ishikawa. This is a brainstorming tool in which a team lists possible causes of the problem by theme (such as operational, electrical, people, instrument, measurement, environment). The team then works each idea back until it finds the one that is the root cause.
- Apollo. This methodology evolved from the investigation of the Three Mile Island accident and expands on the Five “Whys.” Using this method, each response to the question “Why?” is assigned at least one action and
- Shainin system. This method, widely used by automotive manufacturers, tests the RCA result in controlled laboratory conditions to prove it was the root cause. It also tests the proposed solution under the same conditions.
- Other methodologies include cause and effect, event timeline, fault tree, and physical testing of components.
one condition. Once the root cause had been determined, the conditions responsible are controlled to prevent the actions.
Mechanisms to Improve Reliability
Armagost discussed the flow of information and action through the industry after an RCA is completed (Figure 4.4). The various industry players, including API’s multi-segment task group and bolting work group, the International Association of Drilling Contractors/International Association of Oil and Gas Producers BOP reliability group, and the manufacturers, are working to improve bolt reliability. When a failure occurs, an industry-wide alert is issued, usually by BSEE or
the manufacturer. An RCA is launched to establish the root cause, and a bulletin is issued so that equipment owners can take corrective action. Manufacturers or operators implement modifications to a bolt’s design, creation, or operation, and the corrective action becomes standard practice, as was the case with the voluntary replacement of bolts and BSL3 upgrades that BSEE’s Safety Alert 318 recommended. Eventually, it is included in an update of the official industry standards.
The variety of current initiatives to reach these goals illustrates the strong commitment API and BSEE have to improving existing standards, developing new standards, and implementing any necessary replacements. To improve reliability, Armagost suggested that the industry must also continue to self-monitor these steps and conduct audits to verify the necessary modifications.
The BOP reliability database Fleece described is another avenue to improving critical safety equipment. It has broad support from the oil and gas companies, drilling contractors, and equipment manufacturers. There are approximately 2,750 entries with up-to-date failure information for connector bolts, bonnet bolts, hinge bolts, and other BOP components. All entries include the problem, its assessment, a root cause determination, and proposed solutions. If the failure resulted in loss of well control, loss of pressure integrity, or unplanned BOP or LMRP surfacing, or if it was a systemic one, a formal RCA is required.
An RCA is also required if the failure was due to “inadequate component design or configuration,” such as human error or an error with maintenance, manufacturing quality, wear and tear. The root cause will be collaboratively confirmed by operators, contractors, and OEMs and entered into the database. Armagost praised this database as a “wonderful vehicle” to encourage the industry to share knowledge of critical failures and improve communication.
Manufacturers’ continuous improvement processes also improve reliability. Manufacturers who find a nonconforming part can use continuous improvement processes to identify, correct, and contain the problem in order to stabilize the situation and determine the level of criticality. The most critical issues are addressed the most rigorously, and an RCA is required. If the issue is of medium criticality, a nonconformance assessment is undertaken. Low criticality issues are tracked and closely watched for trends.
In closing, Armagost praised the companies, contractors, and manufacturers who make up these working groups and who are pursuing nontraditional collaborations to improve bolt reliability. In particular, he expressed appreciation that players within the industry are willing to hold themselves accountable through voluntary bolt replacement, monitoring, and information sharing via the failure database. Reliability will also improve as manufacturers apply updated standards to their processes, he noted.
There is room for improvement, however. Developing tailored RCA methodologies and expertise, at a level on par with the industry’s technical expertise, could
spur more rigorous evaluations, he suggested. It is also imperative that the industry respond to root causes that require improved practices, equipment, and standards. Armagost closed by noting that the conversations enabled by this workshop are a step in the right direction.
Tom Goin, U.S. Bolt Manufacturing, Inc.
Tom Goin, president of U.S. Bolt Manufacturing, Inc., has experience in many aspects of bolt manufacturing, including sales, supply chains, quality assurance, quality control, and design, especially for critical safety services. He has worked with critical bolting used in the nuclear industry, the Navy, and aerospace, and on oil and gas exploration and production equipment. He serves as chair of API 20E (Specification for Alloy and Carbon Steel Bolting for the Petroleum and Natural Gas Industries) and API 20F (Specification for Corrosion Resistant Bolting for the Petroleum and Natural Gas Industries).
His presentation focused on how API’s 20E and 20F specifications were created to control the manufacturing processes for critical offshore equipment bolts, specifically in regard to forging, heat treatment, machining (or threading), and testing.
Creating New Specifications
API has been writing specifications for almost 100 years, covering every piece of equipment in use except bolts, which are covered instead by ASTM standards for material specifications. However, Goin noted, these standards cover very little in the way of quality control of a material’s manufacturing processes.
Heavy service bolts used in critical operations cannot be reduced to the lowest common denominator, Goin said. Instead, this subgroup of critical bolts has historically been held to a different standard, one that was similar to 20E BSL specifications. As bolt failures started to occur, a consensus grew for API to create specifications that would incorporate best practices for this subgroup to minimize failures on critical safety equipment.
In the past few decades, bolt and steel production has moved out of the United States to countries where these materials can be manufactured at lower cost. Goin asserted that in his view, this combination of cheap materials turned into cheap bolts does not meet the quality standards required for the oil and gas industry’s critical applications. For example, he pointed to a Brazilian company that cut costs by adding boron to steel when manufacturing their bolts, causing hard bands to form in the material. This in turn led to HE and bolt failure.
That example was one of several that convinced API to pursue bolt specifications, and 20E and 20F were created by a task force that examined past bolt failures and detailed specifications to avoid them. The task force paid specific attention to the manufacturing processes, especially forging and forming, heat treatment, threading, and testing.
Process Controls: Forging
Forging control is critical. Goin shared an example of bolts that were otherwise well made but overheated during forging, which caused micro cracks that went undetected during routine testing but expanded under extreme torquing. The cracks were revealed when the rig operators raised the riser up in preparation for a hurricane. Ten percent of bolts on the riser had cracks through their entire head (Figure 4.5).
API addressed this issue with the heating parameters in 20E regarding forging equipment, monitoring and managing of the heating method (especially if using induction heating, which easily overheats material), temperature control, length of time, and dimensional control.
Together, these parameters can prevent microtears, folds, or seams that can lead to cracking and mechanical failure. Goin emphasized that it is essential to control these elements during manufacturing, because after the bolts are made, it is very difficult to determine their fitness for service without destructive testing.
Process Controls: Heat Treatment
20E targeted heat treatment because it can be the source of many problems. As other workshop presenters mentioned, the material for critical bolts should not be piled together or tightly packed into a basket during heat treatment and quenching. Instead, it should be stacked individually with space around each piece to ensure homogenization (Figure 4.6). This careful practice is inefficient for the furnace load and not economical, but it does give a much more reliable and uniform heat treatment and quenching.
20E addresses heating equipment parameters, with special considerations for BSL3 products. BSL3 products can’t be made with induction heat treating or direct resistance heat treating, because although they are the most economical and widely used practices, they are more likely to have problems in their execution that lead to nonuniform material. Cooling media is also controlled, including parameters such as chemical makeup, quench temperature, and agitation method. The final piece of heat treatment is the transfer time, which Goin said must be short so that the material doesn’t lose its temperature while being moved from the furnace to the quench tank.
Process Controls: Threading
Threads must be created properly to prevent internal fractures. While other workshop participants argued that rolled threading is always preferable, Goin disagreed, arguing that there are many different threading methods that can result in the same hardness bands and structures. He pointed to one example in which a rolled-threaded bolt cracked. In that case, the root cause was a misalignment in the rolled-threading machine that created transverse cracks that snapped when the bolt was torqued (Figure 4.7).
20E’s threading standards list several variables, such as single-point cutting, rolled threading, and cutting and forming taps and nuts, for the small-batch production of BSL2 and BSL3 bolts. These variables are not required for less critical bolts because certain drilling conditions, such as shallow gas wells, require far looser specifications than bolts in BOPs 12,000 feet below sea level.
Process Controls: Testing
The 20E specifications also require process verification through metallurgical and mechanical testing, and the latest standards dictate that the test facility must comply with ISO standard 17025. While testing is essential, Goin noted that it has its limitations. Lots are tested after heat treatment to demonstrate accordance with standards, but it’s impossible to destructively test every single bolt. That means that if there has been inadequate processing anywhere along the way, there may be bolts that don’t conform to regulations. Conversely, testing samples of a batch
may uncover a few bad bolts, but that doesn’t mean that every other bolt is bad. In addition, testing doesn’t necessarily reveal all of a bolt’s defects. Hardness testing, for example, only measures the location where the measurement is taken, and as previous workshop presenters noted, hardness can vary throughout a component.
In closing, Goin expressed his firm belief that thorough process control is the only way to ensure bolt quality. Because it is impossible to inspect or test quality into a bolt after it is made, it is, in his view, vital to focus on controlling the process of making it.
Peter Bennett, Pacific Drilling
Peter Bennett is director of subsea support for Pacific Drilling Services, Inc. He has been active in drilling operations around the world, including onshore, offshore, and subsea operations. Bennett discussed the overall importance of bolt failures and reliability and detailed the strategies his company has adopted to reduce risks.
Putting Problems into Perspective
Overall, Bennett said, the industry-wide collaboration to identify critical bolts and implement more rigorous standards substantially reduces risks and is the best safety strategy right now. While risk is a part of daily life, we can and should make improvements when accidents happen; this has been the impetus for stakeholders within the oil and gas industry to join API committees, attend meetings, and convene workshops like this one.
However, although the concerns over bolt failure are warranted because these failures can have high consequences, Bennett cautioned that it is important to put these problems into proper perspective. The voluntary recall of 10,000 bolts wasn’t because they were all bad, but because it was impossible to trace which ones might be bad due to failure to follow the OEM heat-treatment procedures. Of those 10,000 recalled, fewer than 50 were flawed. Therefore, even though failures have occurred and there are clearly problems, there is not necessarily a “pandemic” of failing bolts, he said.
Bennett also pointed out that bolt problems happen throughout many industries, including the Navy, the nuclear industry, and the construction industry. The majority of failures, he emphasized, result from improper procedures, poor maintenance, failure to recognize signs of stress, or pressure to cut corners. These are all ultimately human factors, which can be hard to control. In such situations, awareness of the potential consequences is essential to creating a safe environment. In fact, Bennett said, Pacific Drilling has a “zero tolerance” rule for critical areas; failure to follow procedures results in employee termination.
Pacific Drilling Today
Pacific Drilling currently has a fleet of seven ultra-deepwater drill ships equipped with subsea BOPs. These are delivered from the shipyard equipped with a single BOP stack, and on some vessels Pacific Drilling has installed a second BOP post-delivery. The OEM builds the BOPs from individual components designed, built, and tested according to current API standards. The BOP stacks are assembled and tested by the OEM before they are shipped to the shipyards, where they undergo further testing before deployment.
Pacific has seven vessels in its fleet and 11 total BOP stacks in service. Although the company has not itself experienced any critical bolting failures, after learning about failures on other companies’ equipment, Pacific replaced all of its in-service riser and wellhead connector bolts. Pacific was also part of the recall of a combined 10,000 bolts, which required, in some cases, the suspension of wells and bringing BOPs to the surface, causing significant production delays.
Strategies to Improve Reliability
After the recall, Pacific identified all critical fasteners on its in-service equipment, including upper and lower BOPs, LMRPs, and lower stacks, and undertook several steps to ensure their reliability (Figure 4.8). Engineers pored through the OEM’s data books to verify that all the bolts complied with the relevant manufacturing standards (which they did). The company also altered its procurement system to specify that critical bolt replacements will comply with API 20E.
Bennett stressed the importance of controlling torque. Although OEMs have a certain level of torque control at the factory, Pacific Drilling is responsible for torquing practices in the field, including the correct pattern, number of passes, and approach to re-torquing after pressure tests. As part of its overall strategy to improve bolt reliability, Pacific strengthened its torquing instructions and made efforts to ensure that all employees have the training specified for each piece of equipment they service and operate as required by BSEE’s Well Control Rule.
BSEE also mandates fully transparent failure reporting and documentation of root cause analysis efforts, a requirement for which Bennett expressed his support. In addition, Bennett said Pacific Drilling is a founding member of the group that created the BOP failure reporting database mentioned by other workshop speakers. Pacific and the other companies involved value the database as a way to share knowledge about failures and improve performance.
Bennett said Pacific Drilling is being proactive about bolting failure now, even while the industry awaits new developments from the working groups. For example, the company is voluntarily inspecting all critical bolts through a process known as Phased Array Ultrasound Testing (PAUT). PAUT can’t determine hardness, but it can show whether there are any flaws in a component in service. PAUT was verified over several years by an OEM as a viable testing technique, which means that the results have a good level of confidence. So far, no flawed bolts have been detected, he said.
In closing, Bennett reiterated Pacific’s continued support of industry and BSEE initiatives to reduce failures through improving bolt reliability, quality, processing procedures, and traceability. In addition, he said Pacific is committed to following OEM recommendation and alert bulletins and will only install replacement bolts that meet API’s 20E specifications. Expressing his belief that transparency with peers, clients, and manufacturers is essential, Bennett affirmed that Pacific Drilling will promptly and accurately report failures to BSEE and record them in the failure database.
The workshop’s third panel was moderated by Roger McCarthy, a well-known failure analysis expert who has investigated disasters such as the Exxon Valdez spill and the bombing of the Murrah Federal Building in Oklahoma City. He is founder and president of McCarthy Engineering.
The session’s speakers presented on BOPs, EAC factors, root cause analyses, bolt manufacturing processes, and various strategies to improve reliability. Fleece discussed BOPs in detail, including their equipment, failure experiences, a collaboration to share BOP failure knowledge, and BP’s current bolt action plan. Adamek emphasized the need to spend more time examining unknowns related to the environment’s role in EAC, particularly with regard to CP. Armagost covered root cause analyses and how their findings can improve reliability. Goin detailed his role both as a bolt manufacturer and as the chair of the API group that created the standards for critical fasteners, 20E and 20F. Bennett described the on-the-ground work that his company, a drilling contractor who owns several rigs, is doing to improve reliability.
McCarthy moderated a lively, wide-ranging discussion following the panelists’ presentations. Workshop participants and panelists addressed details of the 10,000-bolt recall and circled back to other issues related to bolt manufacturing, considered the role of human factors, and discussed environmental factors including impressed current systems.
Lessons from the 10,000-Bolt Recall
Prompted by a question from McCarthy, participants discussed the particulars of the recall of some 10,000 H4 connectors and considered what might be learned from that experience.
Adamek noted that the bolts that had cracks had been zinc plated but not baked in accordance with ASTM’s standard B633. Bolts that were baked in accordance with B633 were not recalled. Rather than waiting for the root cause analysis, the industry decided to recall the lots that might have been improperly treated.
Out of 10,000 bolts recalled, Adamek said about 3,000 to 4,000 bolts were returned. A thorough inspection revealed no cracks in the recalled bolts, an outcome Adamek described as surprising and distressing, since, although it is known that roughly 66 bolts on three separate vessels failed, it remains unclear why some failed while others did not.
Adamek added that all the bolts examined showed some embrittlement. At the same time as the recall, another failure occurred shortly after a drill bit had become stuck and two days of powerful jarring was required to unstick it. The failure report from that event concluded that the bolts were already cracked, but this extreme
jarring was a secondary effect that led to the failure. Bennett clarified that when the BOP components were recovered for testing, at least half of the bolts were still intact despite the embrittlement and the jarring.
As news of the recall made its way through the supply chain, Adamek said, a supplier pointed out a conflict between two ASTM standards, noting that one standard required baking and the other didn’t. Clyde Briant asked how many zinc-plated bolts failed, and how many bolts were zinc plated but were not properly baked and didn’t fail. Haeberle recalled that all of the bolts on the affected GE BOP connector were improperly baked, but referred to BSEE’s QCFIT report for details on how many bolts actually failed.
Bolt Design and Manufacturing Processes
One workshop participant offered additional context on the design of bolts. H4 connectors (the type implicated in the 10,000-bolt recall) were originally designed in 1964, before the need for hardness limitations was understood. H4 has evolved several times and is still in use today. Engineers are experimenting with ways to reduce hardness—to 32 HRC, for example—in an effort to improve the safety margin. The participant also noted that the DWHD Bolting, which failed on the Discoverer India rig, was totally redesigned in response to its failure and the bolts it required also changed.
David Matlock asked Goin for further details about heat treating, in particular how a “lot” is defined in this context. Goin explained that a lot refers to material that is from the same heat, is of the same diameter, and is heat treated together as a batch. As an example, he explained that if there are 100 tons of a material heat, but a manufacturer can only heat treat 2,000 pounds at a time, then there are 100 lots. 20E does not prescribe the exact amount of material that can be heat treated together; instead, this variable is determined by the manufacturer’s abilities, heat-treating source, and quenching equipment.
Participants then turned to focus on the issue of zinc coatings. Adamek explained some of the rationale behind the use of zinc. At a 5-year equipment recertification test, every component, including bolts, is disassembled, cleaned, inspected, and recoated. During this process, it was found that bolts coated in phosphate were corroding too quickly, so engineers applied zinc in order to increase their life span. On one rig that had several bolts fail, a BOP connector (the piece of equipment, not a bolt) also failed, and the bolts on that equipment, which were coated in phosphate, had no cracks. On the LMRP failure, however, there were zinc-coated bolts and they did crack. Although the cracking was attributed to EAC, Adamek said more study would be needed to determine the exact cause.
In response to a question from Amaya, a participant clarified that the bolts failed 2 to 3 years after the zinc plating was applied. Participants debated whether
the zinc plating had completely disappeared at the time of failure. Adamek clarified that further research determined that the bolts were zinc plated, installed in a BOP connector, and torqued to 100 percent of their yield. Adamek said in his view, although zinc plating probably introduced minimal hydrogen, the fact that the bolts failed after several years suggests that there were other contributing factors at play.
Building on the notion that zinc may not fully deserve its negative reputation, Haeberle pointed out that zinc phosphate and zinc plating are distinct coatings, charge very different levels of hydrogen, and react differently to the environment. Zinc plating is a sacrificial coating, but zinc phosphate is not, he said. In response to a question from Ian MacMoy, Haeberle pointed out that anodes are providing protection in the exterior environment, and hydrogen is generated on the seawater side. On the Enterprise rig, bolts were inserted through flanges in such a way that seawater was present inside and CP wasn’t protecting the area, and a galvanic cell was in the wrong area.
Testing and Inspection Methods
McCarthy asked what size flaws are detectable with the PAUT method. Another participant responded that after several adjustments and practice tests in the lab and on the rigs, testers are able to detect flaws at a millimeter or less with 95 percent confidence.
John Scully asked for more details on the bolt recall, specifically, how the unbroken bolts were inspected. He pointed out that cracks don’t mean a component will snap and break. Were the testers looking for incipient cracks, or residual hydrogen, or other failure signs? Leo Vega, Stress Engineering, answered that in a metallurgical investigation, testers examine areas where cracking indications are most likely to be, usually the nut and the flange interfaces. In the case of the recall, inspectors also looked at the coating and found microcracks there and in the middle of the head. He explained that inspectors use the least intrusive process, ultrasonic cleansing, to clean the bolts and then light media brushing, wire brushing, or light media blasting to find the cracks. The location of the cracks is then reported to GE. Haeberle noted that after the cleaning, inspectors also did a standard mag particle examine, a method used for testing new bolts.
Brun Hilbert asked how often BOP stacks are inspected via ROV. Fleece responded that these inspections are performed every 3 days, at a cost of about $10,000 per day. There may be two ROVs, each with eight crew members, and that is part of why deepwater rig drilling operations are so expensive, he said.
Narasi Sridhar pointed out that Fleece’s ROVs inspect drilling equipment. What about production equipment? Fleece said that he does not have experience with production equipment, but noted that subsea pipelines and wells have many safety valves, some independent, to provide redundancies and mitigate spills in
case a pipeline loses integrity. Elliott Turbeville, TechnipFMC, added that some production equipment never gets inspected, although it is designed to be under water for 20 years. Sometimes equipment is inspected after it is removed at the end of its life span, but not in all cases.
Hilbert asked whether all the bolts are changed out during the major 5-year inspections. Fleece replied that it varies, but most testing is done in situ or by proxy, and there are some rigs that are disassembled and rebuilt every 5 years. These inspections are also a good time to replace critical fasteners in place before API 20E with lower-risk, lower-hardness bolts that comply with 20E, he noted.
Turbeville asked if voluntary replacement of 20E-compliant bolts has led to any operational changes, since higher-strength bolts are replaced with lower-strength ones. Bennett replied that a careful examination of processes revealed that there were only small changes needed, including changing a few other components, reducing yield strength and bending, and slightly increasing the time it takes to shut down in the case of a large storm. Fleece weighed in to note that critical bolts operate under a variety of different yield pressures, and changing bolt hardness and reducing yield often doesn’t have a large operational impact.
Matlock asked Turbeville if it was true that his company does not use high-strength bolts. Turbeville said yes, but noted that while TechnipFMC makes nearly everything else in use in the subsea environment, the company doesn’t make BOPs. It typically uses the 80 or 105 specification, for sour service or seawater compliant bolts, which have a hardness of 22 or 34 HRC, respectively.
Addressing Human Factors
Nancy Cooke noted that several speakers mentioned a failure to follow procedures or specifications, either because employees don’t know they exist, can’t access or understand them, or find them too complex. She asked for the panel’s opinion as to why people aren’t following procedures.
Bennett confirmed that all the issues Cooke listed are a concern and added that it is important to make it as easy as possible for people to follow proper procedure, especially for unwieldy processes. One effort to address this problem is that procurement requests must now include the relevant wording from 20E.
Following up on this point, Adamek noted that for zinc plating, an experienced vendor interpreted the standards differently (but accurately) and made bolts according to the 1998 specifications instead of a more recent version. Unfortunately, these specifications allowed for zinc plating without post-baking, and once this interpretation was uncovered, the recall was issued. This problem illustrates how important it is to control the entire length of the supply chain.
Fleece said that BP is working to improve personnel competency. Subsea BOP support staff may not always have adequate training or certification to maintain,
service, and inspect the equipment, and that needs to change, he said. Right now, third-party verifiers often document the work and maintenance on a BOP or rig to ensure implementation of proper procedures, routines, and OEM recommendations. While these systems act as safeguards, Fleece said, there are broader issues that add complications to every task. Noting that the level of collaboration that resulted in the failure database, the work groups, and the task groups did not exist before 2015, he said persistence is now paying off, and collaboration and information sharing among companies is likely to help the human factors issue in the future.
Cooke asked if following procedures required specialized skills because of their complexity. Fleece replied that it varies, and while some operations do require highly technical skills and specific operations, for others, BP tries to “drive out that complexity” to ensure that most rig workers could read a procedure and implement it properly. Bennett added that if a written report is involved, there can be a great deal of variation in the quality of writing. Another issue is terminology; to address this, the failure database uses drop down menus to ensure consistency. Checklists, as used in medicine and aviation, can also be useful for eliminating inconsistencies.
Jon Shoemaker noted that the Health and Safety Commission in the 1980s created a robust system for reporting and reducing injuries, which increased safety on the rigs. Previously, “failures” merely meant any equipment downtime resulting in reduced profits. Today, we are instead focusing on every failure, and if a failure is not technical or specification-related, then investigators look to human factors. When evidence of repeated behavior patterns emerges, we should examine our procedures to ensure workers have the right tools, Shoemaker said, adding that in his experience, procedures have to be written by people with on-the-ground, operational savvy. For torquing, for example, Diamond Offshore specifies and certifies all the equipment needed, including torque load cells, which can simplify the process.
Environmental and Operational Factors
Scully pointed out that the industry has made good progress in understanding the importance of material quality and stress management. As some of the panelists mentioned, the one area we can turn to next is a better understanding and control of the environment. He raised several questions in this arena. First, do the people in charge of the critical fasteners communicate with those in charge of CP? Second, who has final control over that issue? And third, is it realistic to continuously monitor the CP current?
Chris Johnson, National Oilwell Varco, works at an OEM. During the stack design process, he said his company conducts an analysis to determine the best placement for anodes, and then a third party verifies the location, number, and assembly for proper protection. Typically, the fastener specialists are not involved
in this discussion, but because the overall goal is proper protection, operators use no more anodes than are strictly needed in order to keep the current stable.
Eric Larson, GE, stated that GE has a research center that develops design practices and calculations, instead of a third party, to determine the best coverage. GE, he said, uses DNV’s CP standard to calculate anode mass and draw over time. In his experience, the same people who do these calculations are the ones who install BOP bolts, so there is collaboration, but it’s always possible to improve design practices in the future.
Russell Kane asked Bennett for his perspective on ICSs on ships, and specifically their operation, maintenance, monitoring, and performance relative to the ship and the riser. Bennett expressed surprise at Adamek’s theory of increased use starting in 2000, because in his past experience as a marine engineer, he encountered ICS use as far back as 1979. He indicated he would make inquiries in order to respond more fully.
Kane then asked Bennett if he knew why his company hadn’t experienced critical bolt failures. Bennett responded that his is a relatively new company with only 11 BOPs, and the company has only just started its first 5-year maintenance cycle. He also pointed out that the failures have been isolated, and so there are probably many other companies that have had no failures.
Robert Turlak, Transocean, noted that his company uses ICSs and has had one riser problem (on the Enterprise) and one connector problem (on the Discoverer India). Kane asked where the CP measurements were, and Turlak replied that they have several measuring devices on the LMRP. Kane agreed that impressed current probably couldn’t travel farther than 6,000 feet, but expressed concern about CP behavior, noting that in his experience, anode consumption in deep water is quite high, which means the current is being pushed somewhere else.
Fleece added that Transocean is the largest drilling contractor in the world, and so its failure level is a matter of statistics—it has the most rigs. Bennett’s company is small, so it makes sense, from a statistical perspective, that it hasn’t had any failures. The failure database has shown that any rig anywhere in the world can fail, even though they are all from the same OEM, are made of the same materials, follow the same set-up procedures, and are in similar service conditions, Fleece said.
Sridhar asked if the failure database information is available to the public. Bennett answered that it isn’t, but that BSEE has access to it, and the data that fall under the purview of the Bureau of Transportation and Statistics (BTS) is publicly available. Fleece added that the BTS performs defect analysis for BSEE from research in that database. Adamek noted that BSEE or BTS will provide a report
at the Offshore Technology Conference and then release a formal report that will include some of that data.
Jenny Mandel, E&E News, asked how much of the information from the working groups and the 2017 short-term recall of bolts greater than 35 HRC will be publicly available. Holly Hopkins, API, responded that most of what API sends to BSEE is publicly available on the BSEE website. Mandel noted that she had trouble identifying it on that website, and Hopkins offered to help Mandel locate the information. Candi Hudson, BSEE, added that although BSEE is currently revising its website, many reports should still be accessible, including API’s progress reports; information about task group activities; and information on factors like CP coating, load, fatigue, and manufacturing. Mandel clarified that she is most interested in learning about any new, immediate actions the industry is committed to taking to directly address the near failures, as opposed to the deeper, long-term discussions such as those that have been the focus of this workshop. McCarthy noted that all of the information from this workshop will be publicly available.