HOUSTON — Richard Lynch was walking down the hall in BP’s crisis command center in early May when some engineers rushed up, bearing bad news.
“We’ve lost the cofferdam,” they said.
In fact the cofferdam, a 100-ton, four-story-high steel dome that the company had lowered to try to contain the flow of oil from its out-of-control well, had become clogged with icelike crystals and was rising in the water, full of flammable gas and oil.
“I said: ‘What the hell do you mean you’ve lost the cofferdam? How did you lose it? Don’t give me that!’ ” Mr. Lynch, a BP vice president and a leader of the effort to kill the well, recalled. “This thing has taken off like a damn balloon.”
Had the dome hit one of the work ships, another inferno like the one that destroyed the Deepwater Horizon drilling rig might have resulted, with more lives lost. But eventually the engineers managed to maneuver it to safety.
“The last thing you’d want is this thing filled with ice rushing up to the bottom of the vessel,” Mr. Lynch said.
The official death of the now-notorious Macondo well in the Gulf of Mexico is expected after Labor Day, with the completion of a relief well. Whether the four-month effort to kill it was a remarkable feat of engineering performed under near-impossible circumstances or a stumbling exercise in trial and error that took longer than it should have will be debated for some time.
But interviews with BP engineers and technicians, contractors and Obama administration officials who, with the eyes of the world upon them, worked to stop the flow of oil, suggest that the process was also far more stressful, hair-raising and acrimonious than the public was aware of.
There were close calls, the details of which were not released to the public, like the panic over the rising dome. Sleep-deprived men and women neglected family birthdays and watched long-planned summer vacations vanish. Inside the command center here and at the well site, 40 miles off the coast of Louisiana, tempers flared — in one heated argument, a senior engineer on a ship threatened to throw another senior engineer overboard — and blood pressures rose.
The dome was only the first public debacle. As failure followed failure, the relationship between BP executives and administration officials deteriorated, resulting in disputes that some oil industry experts say delayed the killing of the well.
Looking back, administration officials said that they became concerned that BP could not handle the crisis and that at crucial junctures the company made serious errors of judgment. “There was an arc of loss of confidence,” said Interior Secretary Ken Salazar. “I was not comfortable they knew what they were doing.”
Those on the industry side saw it differently. “The only benefit I see is they actually challenged us to a level of detail and communication,” Mark Mazzella, BP’s top well-control expert, said of the government scientists who stepped in to supervise the effort. “They didn’t offer anything that changed anything we actually did.”
A decision by Energy Secretary Steven Chu to turn to BP’s competitors for advice was viewed as an insult by many at the company, said a technician who insisted on anonymity because he was not authorized to speak about the matter.
The tensions above filtered down to the command center — a series of rooms on the third floor of a tower on BP’s campus in Houston’s Westlake section — where the seasoned well experts faced a challenge of unparalleled scale and difficulty: to apply techniques used often on land to an out-of-control well 5,000 feet underwater.
“If ever there was a ‘war,’ this would qualify,” one contractor wrote in a letter to BP.
The walls of the center were papered with notes from well-wishers: “You are heroes, may God lead, direct and protect you all,” read a poster sent by BP workers in South Africa. BP offered the scores of exhausted workers services including massages, “stress therapy” and information on sleep hygiene and ergonomic techniques to reduce physical strain.
But by mid-August, when the oil was finally contained, the well was cemented and all that remained was the completion of the relief well, the intensity of the experience was becoming clear to many of those who worked to stop the worst marine oil spill in American history.
“It’s been one of those sort of very profound periods in your life,” said Paul Tooms, BP’s vice president for engineering. “I’m not quite sure what normal is going to feel like after this.”
Mr. Mazzella, the BP well-control expert, who ropes and rides at rodeos in his spare time, was at a practice rodeo in early June when the wife of a friend confronted him.
“You’re not doing a very good job, are you?” she said.
Even Mr. Mazzella’s elderly father pointedly asked him one day when BP would finally get the well plugged.
“We are tired of hearing about it on the TV,” he told his son.
Mr. Mazzella said he and his colleagues struggled to shrug off the criticism and stay focused on their task. But the humiliation peaked over Memorial Day weekend, when the procedure called the top kill also met with failure.
Senior BP executives and government officials, once again, had publicly offered optimistic predictions about the success of the technique, which involved pumping in heavy drilling mud and, in a process known as a junk shot, assorted objects including golf balls. Privately, it turned out, some engineers with BP and with Wild Well Control, a contractor, were far less confident that the top kill would work.
For three days, engineers worked high-powered pumps on two surface ships to overcome the oil and gas belching out of the well.
At one point, technicians said in interviews, a plumbing problem on one of the pump ships forced a delay in the operation. Then a screaming match over the radio between two senior engineers ended in one of them threatening to come over and throw the other overboard.
At the Houston command center, officials assembled to monitor the top kill. A BP technician called out pressure readings. Dr. Chu, in shirtsleeves, performed his own calculations with paper and pen.
As they watched, the pressures started to decrease — a sign that the pumped-in drilling mud was succeeding in overcoming the pressure of the oil spewing out of the well. There were high-fives around the room, and government officials sent text messages to the White House saying that victory might be near.
But an hour later, the pressure readings leveled off. The attempt had failed.
The next day, Dr. Chu, concerned about putting too much pressure on the well, ordered an end to the operation. It was a turning point: the government was now in charge, and with greater frequency, Energy Department officials and scientists were conferring with Exxon Mobil and Shell engineers, asking for advice about what to do next.
For BP’s engineers and technicians, it was one more thing to be depressed about. Mr. Mazzella recalled the faces of his crew members when they returned from the ships by helicopter to Houma, La., after the top kill had failed.
“They were down, they were,” he said. “It impacted everybody when we had to walk away from that thing and the oil was still flowing.”
“Everybody came up to me,” Mr. Mazzella said, “and they were almost apologetic — ‘We’re sorry we didn’t get this done.’ ”
His voice trailed off. “We didn’t get this done,” he repeated softly.
Learning From Failures
With the top kill abandoned, “it was quite obvious that this was going to keep going for some time,” Mr. Tooms recalled. The problems facing the team seemed overwhelming.
“I didn’t feel like I had the answers,” he said.
For the most part, Mr. Tooms said, he was able to ignore the news clippings sent by friends back home in Britain. But reports in the American news media would send him ranting to his boss.
At the same time, government scientists were starting to press BP for more data and more analysis.
Dr. Chu told his Energy Department associates he was no longer willing to settle for half-measures or wishful thinking. “I wanted to make sure this thing is really killed dead, dead, dead,” he said in an interview.
But Kent Wells, a BP senior vice president who early on became the effort’s public face, said in an interview that looking back, the Houston engineers had learned something from the failures. The experience with the cofferdam, for example, had taught them a better way to cap the well — avoiding the formation of the icelike crystals by first lowering the capping device to the seabed off to the side, away from the plume of oil, then sliding it into place. This paid off less than a month later, he said, when engineers installed a loose-fitting “top hat” cap on the well.
The top hat was the first modest success, eventually funneling about 15,000 barrels of oil a day for surface collection. But it was viewed only as a stopgap; what was needed, many on the team were convinced, was a more radical approach, one that had been proposed only a few days after the blowout on the Deepwater Horizon rig, when the team brainstormed solutions to the disaster: a tight-fitting cap.
“To meaningfully go forward in any rational way required a pretty bold step, which was to open up the flow again and put a device on that that would give you some pressure control,” Mr. Tooms said. “That was for me the defining moment.”
A Pressing Question
As bolts go, this one was enormous: nearly a foot and a half long and tipping the scales at 51 pounds. With five identical ones, it held what remained of the broken riser pipe atop the well’s crippled blowout preventer at the seabed.
If there was to be any hope of putting on the cap and shutting off the flow of oil before the relief well did the job later in the summer, the bolts had to be removed. But no one knew how stuck they might have become by sitting in the deep for so long.
To the public, their expectations dulled by the repeated failures, a tight-fitting cap was just one more hill on a four-month roller coaster ride. But engineers were focused on a single pressing question: Could the bolts could be loosened by remotely operated submersibles, the high-tech marionettes that did the work in the crushing pressures and frigid temperatures 5,000 feet down?
The engineers scrounged around and came up with the biggest subsea torque wrench they could find, and watched from the Houston command center as a remotely operated submersible used it to easily loosen one of the bolts.
That seemingly simple act was a game-changer, said Mr. Lynch, the BP vice president.
“Suddenly,” he said, “I’ve got a pressure containment device I can put on this, and it’s real and it works. Now I’ve got an opportunity to close the well in.”
Which is what the engineers did several days later, bolting on the new device, called a capping stack, and preparing to conduct an “integrity test” by slowly closing valves on the stack and raising the pressure in the well.
Yet the test was delayed, a fact that BP and government officials publicly attributed to a benign request by the government for more information. In fact, a dispute had erupted. BP wanted to go ahead with the tests. Dr. Chu and his advisers were blocking them. Closing the valves, they argued, could force the oil out of the well and make a bad situation much worse.
The government team convened a conference call with hydrologists and geophysicists from universities and other oil companies. They raised alarms to Mr. Wells, Andy Inglis, BP’s chief executive for exploration and production, and James Dupree, a BP senior vice president, who continued to insist that the procedure was safe, administration sources say.
“Chu raised his concerns about the subsea geology with the BP people and they couldn’t answer his questions,” an aide to the energy secretary said. “The result was that the plan to conduct the integrity test was halted for 24 hours.”
Some BP executives, a government official said, were furious, skipping some of the scheduled engineering meetings over the next several days.
“It was like, ‘Where the hell are they?’ ” the official said. “It was frustrating to them that people were still asking questions.”
Eventually, BP engineers persuaded the government to continue with the test, and on July 15, Mr. Tooms stood in the engineering room, watching the valves being closed on video feeds.
The room was hushed, except for the background radio chatter between technicians in Houston and out in the gulf, calling out pressure readings and the number of turns as the final valve was dialed closed. “And then there was almost kind of a pause,” Mr. Tooms said, “kind of like a realization that it stopped.”
At two news conferences on July 16 — the tension between the government and BP had long before led them to hold briefings separately — Mr. Wells and Thad W. Allen, the retired Coast Guard admiral who led the federal government’s response to the spill, announced that the test had started. Oil was no longer flowing into the gulf. On BP’s live video feed, the torrents of oil and gas gave way to images of small white particles drifting lazily through the water past a quiet metal hulk.
Yet how long the respite would last was uncertain. The government scientists were still skeptical about leaving the cap closed, while BP engineers were convinced that the well could handle the pressure.
The issue came to a climax at a meeting the next morning in the 20th-floor boardroom at the BP office tower. Mr. Tooms’s team took up many of the 22 seats around the long conference table. They were joined by government scientists. Dr. Chu was dialed in by speaker phone, as were Carol Browner, President Obama’s adviser on environmental issues, and Admiral Allen.
“We just laid out our technical case of why we believed what we believed,” Mr. Tooms said. “When you have the level of intellectual you are talking to, they are going to ask you questions about why your technical case is correct. And we went through all that.”
It would prove a pivotal moment. After hours of discussion, the government agreed to keep the cap closed. The pressure held, the flow of oil remained shut off and BP could eventually proceed with plans to seal the well permanently.
“Had we opened it up at that point that would have been, I think, probably the darkest moment in my career,” Mr. Tooms said.
One More Twist
In the weeks after the oil was shut off, the well offered up one final surprise.
The story had gradually receded from the headlines — a comfort to BP executives, who had watched the company’s stock price plummet, and to the Obama administration, which had been criticized for the speed of its response to the leak.
But on Aug. 2, as workers prepared to pump mud into the well to kill it permanently, an engineer stuck his head into office of Mr. Lynch, the BP vice president.
“Richard, we’ve got a problem,” the engineer said.
A hydraulic leak had caused a critical valve on the cap to open up again. A second, fail-safe valve behind it was being kept closed by little more than friction. If that second valve opened, oil and gas would again start pouring into the gulf.
In the end, the valve held, and a renewed nightmare was averted.
But as Mr. Lynch recalled, “The difference between us having the well shut and everything going swimmingly well and the fact that we could have been flowing hydrocarbons back into the sea was that close.”