Tuesday, October 02, 2012

Guest Post: Managing an Unprecedented Plant Challenge at San Onofre

Pete Dietrich, SCE
For most of 2012, the nuclear industry has been keeping a close watch on the situation at San Onofre Nuclear Generating Station, where continuing issues with the plant's steam generators have kept both units offline.

As NEI member company and plant manager Southern California Edison (SCE) has been investigating the source of the problem, it has faced a unique challenge with its own employees: how to communicate about the situation when the path forward is uncertain.

In the following guest post, SCE's Senior Vice President and Chief Nuclear Officer Pete Dietrich describes how he and his team embraced this unique set of circumstances. Today's post is the first in an ongoing series concerning the status of the plant.
Part 1 – Aligning the understanding and expectations of a large plant workforce when the path forward is uncertain

At 3:05 p.m. this past Jan.31, control room operators at our San Onofre Nuclear Generating Station detected a small steam generator tube leak from one of our two units. In accordance with our procedures and their training, operators safely shut down the unit and we began the process of determining the source of the leak. You have probably heard what we discovered during the months of inspections and analysis that followed – a previously unknown steam generator tube-to-tube wear mechanism resulting from in plane vibration that here-to-fore had not been experienced in similarly designed steam generators.

We were gratified that U.S. Nuclear Regulatory Commission inspectors found that our operators responded to the unique event in a faultless manner that protected public health and safety, using the conservative decision-making and questioning attitude that are trademarks of the nuclear power industry.

A large team of technical experts we assembled to help us understand the cause of the leak, including some of the best minds in the business, determined that tube-to-tube contact arising from excessive vibration in certain areas of the steam generators caused the unusual wear. We concluded that the excessive vibration was produced by a phenomenon known as “fluid elastic instability” which results from a combination of factors, including steam velocity, moisture content of the steam, and effectiveness of the supports in the areas where the vibration occurs.

One of the leadership challenges we have faced during the months of analysis of our steam generators has been maintaining a common set of understandings and expectations about our future across a large plant workforce as the investigation was evolving. I refer to this as a leadership issue, rather than merely a communications challenge, because the technical resolution path we are traveling is extremely challenging, even in this highly technical industry. As a result, countless decisions have been required of senior plant leaders about how we remain transparent to employees while not speculating about timelines, operating schedules and even outcomes.

San Onofre Nuclear Generating Station
The same has been true of our relationship with the public. How does a plant maintain its commitment to openness with its neighbors when it isn’t clear exactly what the analysis process will find? How do we describe the path forward when it is not yet clear? That was the leadership challenge, but more importantly, an opportunity to be fully transparent to the public as facts are determined and understood.

Had you been present in our daily briefings, one of the tactics you would have heard discussed frequently to meet this challenge is “communicate, communicate and communicate again.” We established very high communication expectations both up and down the normal channels and among the consultants during these months. I directed plant leaders to communicate well beyond previous levels, and to do so in person as much as possible. And those of us in senior leadership did the same, hosting an increased number of meetings with plant leaders and individual contributors for updates and Q&As. We believe saying “we don’t know yet” is better than saying nothing or speculating.

Another tactic we have used to fill what could otherwise be a risky information gap, as we await research conclusions, has been to continuously identify milestones that can be communicated. This is a way of signaling progress, even minor progress, and thereby offsetting the false impression that a slow, methodical, meticulous review process is a faltering process. At each of these milestones we have stated what we know so far, what we don’t know, and when we will have more information.

A unique and related leadership challenge we faced was to maintain schedule discipline in the face of unknown results of the ongoing investigations and research. In this situation the usual schedule driver of plant restart is no longer effective. At the same time, leaving the recovery schedule open-ended undermines confidence. Our solution was to impose team and individual discipline with short-term project schedule deliverables for identified tasks needed to define alternate success paths for safe return to service—and periodically adjusting the overall plant startup schedule to reflect the understandings coming from the individual short-term tasks. Our project team heard our message—getting to the right solution was the real priority, and meeting near-term commitments was the path to get there.

In addition to our efforts to maintain a cohesive understanding among employees about our prolonged analysis process, we have worked to align expectations about the prospective outcomes. For example, we’re reminding plant employees that, no matter the technical outcome, when a difficult problem is managed well, it can make a significant contribution to industry knowledge; and that contribution impacts the way a plant staff is viewed by industry peers. We will succeed by continually focusing on the nuclear safety aspects of our analysis and decision-making.

Our goal remains to conduct a comprehensive and independently challenged analysis of why relatively new steam generators would develop significant tube wear issues that benefits both San Onofre and the industry by gaining a full understanding of this new degradation mechanism. We will create a corrective action plan that is viewed by our peers and the regulatory authority as comprehensive and is based on prudent decision-making that assures safe operation of the plant. In this way, we will help our industry avoid a repeat of what our operators experienced on that Tuesday afternoon in January.
Thanks to Pete and his team at SCE for all of their hard work -- and for putting the safe and reliable operation of the plant ahead of every other priority. We look forward to future contributions.

4 comments:

donb said...

The process to get the San Onofre plants running again is certainly painful and expensive. But as Henry Petroski points out in his books (especially To Engineer Is Human), failure is a necessary part of engineering. From failure we gain much new information. This seems to be as important as ever. We sometimes think that our sophisticated computer models will prevent us from having unplanned failures, but then physical reality takes over and says "Not so fast!" Eventually, we will have even better computer models for heat exchangers of this type so that such accelerated wear can be avoided in the future.

Anonymous said...

Sorry, donb, I don't buy that argument ("failure is necessary"). Given the financial risks, the component design needs to recognize the uncertainties in the methods and ensure failures like this don't happen. The original SG design was not perfect, but it apparently was superior to the RSG design. I think this validates the conservative design approach followed in the late 1960s.

Anonymous said...

Some of the BWRs have experienced similar fluid-elastic instability in steam dryers caused by higher steam velocities after power updates. In these cases, the solution was to modify the steam dryers. In the case of the SONGS steam generators such modifications may not be practical, but it is possible to reduce power to a level where the fluid-elastic instability ceases. This can be verified by shutting the plant down after a relatively short period of operation and confirming that the vibration-induced wear has stopped.

As a consequence of the SONGS outage, SCE has had to restart old, dirty fossil plants. So it is important to move forward to get SONGS restarted.

Ted Quinn said...

The key to the restart is the prudent application of NRC resources in the technical oversight of the restart process. NRC has reviewed many steam generator replacement programs around the U.S. and has the experience and trained personnel to complete this for both San Onofre Unit 2 and 3 with particular emphasis on the restart of Unit 2, which had no structural failures (as stated by the NRC).