How not to terrify an entire state: lessons from the California fire safety power cuts

Many people couldn’t find the information they needed during the recent fire safety power cuts in Northern California, due to poor organization by web search companies and website problems at utility companies. This created confusion and frustration, and could have meant much deeper problems if the outages had lasted as long as originally feared. 

The problems were revealed by user tests conducted during the blackouts by UserTesting. The tests showed that significant improvements in usability and server reliability are urgently needed to cope with future power cuts. The incident has important lessons for anyone involved in public safety. With potential power shut-offs looming for this week for Northern and Southern California residents, addressing the problems can’t happen soon enough. 

During the blackouts, UserTesting surveyed people in affected counties to ask if they had been affected by the cuts, and how they prepared for them. We also captured video of the web resources they used for information, and what worked and didn’t.

Key findings included:

• Awareness was high. The utilities, news media, and government did a thorough job of alerting people to the possibility of power cuts.
• Follow-through was haphazard. However, many people struggled to find specific information on whether they would be affected. Specific problems included difficulty finding the right information through web searchers, and breakdowns in the utility’s online map and service lookup function.
• People were unprepared. Even people who knew they faced blackouts generally failed to do significant preparation for the outage. If it had lasted up to a week, as originally feared, many would have had serious problems.
• Many people were surprisingly understanding, for now. Although there has been a lot of media reporting about public anger about the power cuts, the people we spoke with were generally understanding of the near-term need for the power cuts. However, most were upset that the utility’s infrastructure was so vulnerable to high winds, and some warned that they will be much more upset if the blackouts happen repeatedly. 
• We were lucky the blackouts were short. The power cuts lasted less than a day in most cases. This limited the problems caused by many peoples’ confusion and lack of preparation. If the cuts had been longer, the problems could have been far more serious.

The remainder of this research is divided into three sections:

• How people coped with the blackout
• Lessons for organizations that plan public safety
• Methodology of the user tests

How people coped with the blackouts

Most people heard about the cuts through a variety of media. The most common sources were websites (the local utility, Google, local news sites, and Nextdoor were all mentioned), social networks like Facebook, and conversations with friends. Many people said they expected Pacific Gas and Electric (PG&E), the local utility, to be the ultimate source of truth about the cuts, and expressed frustration when they had to go to other sources to get information.

Sixty-five percent of the people we spoke to were notified that they might lose power, and of those people, about half actually did lose power. One person said they lost power but were not notified about it in advance. Many of the people who were notified complained about the vagueness of the warnings. Many said they were notified only that they “might” lose power, but they wanted specifics on whether the cutoffs would happen, and when, so they could make plans.

Most of the people who were notified, and some who were not, did small amounts of advance preparation. This usually consisted of buying bottled water and non-perishable food, and filling the car with gas. A few did more complicated preparation, like buying a cooler and ice, or buying propane. No one reported that they had prepared for a blackout as long as seven days, even though many of the public notices said to do that.

Because it was short, the impact of the blackout on most people was mostly emotional. A few people reported losing food, and a couple missed a day of work. But many reported anxiety and frustration as they tried to determine if their power would go off, and when it would come back. A couple of people reported that they would have had serious problems if the outage had lasted more than a day. For example, one person depended on daily medication that needs to be refrigerated. He had bought ice and a cooler to hold the medicine, but that was good for only about a day.

Most people reported that they took steps online to find out if their home would be affected by the outage. Most of them started with a web search for “power cuts” or phrasing like that. Some went directly to the website of PG&E. The results of those online searchers were mixed:

• Among those who went to the PG&E website, most of the desktop users were able to find the outage map and determine if their home was covered in it. But some people struggled to navigate the map or understand what it meant to them. The success rate was lower for mobile users. The link to the outage map was less prominent on the PG&E mobile website, and some users had trouble navigating the map on a smartphone.
• Several people remarked that they had looked for the PG&E map the day before, and it was not available because the company’s website was overloaded. So they used other methods to find information. Some people who didn’t find the PG&E map found outage maps on local news sites. A few landed on an outage map created by Google, but had trouble interpreting it because it used icons rather than a drawing of the outage perimeter. 
• Map navigation, in general, appears to be an issue. Some testers struggled at zooming in and out, or interpreting what they saw.
• People who didn’t find a map wandered through news stories, online videos, and other sites but usually could not find definitive information.

Most of the participants acknowledged that the cuts were necessary to prevent fires, but after that their opinions split. Some felt the incident was well handled in general. Many felt the immediate execution of the cuts was well handled, but were angry or frustrated that the power company’s infrastructure was vulnerable to wind problems. A few questioned the need for the cuts at all. Many said they will be less understanding if the power cuts are repeated.

Lessons for organizations involved in public safety 

Initial notification was good, but after that the process often broke down. The broad notification process was effective at letting people know something was going on, but many people struggled to find details that would let them act on the situation. Web searches were often confusing, a key website failed, and the notifications given to power customers were frustratingly vague. This created a lot of anxiety. If this had been a broadly life-threatening emergency, the results could have been serious. Four issues stood out in particular:

• Web search is a key part of an emergency response. When confronted by uncertainty, the first instinct of most people is to do a web search. That puts a lot of responsibility on sites like Google to ensure that the most important information is displayed prominently. Most people were looking for an outage map, but searching for one produced a mass of links, videos, and often a connection to a Google map that was frustratingly vague. If you knew where to look, the right information was available, but many people couldn’t find it. This is a point of potential failure that needs to be addressed.
• Make key information easy to navigate. Some people who did find the outage maps struggled to interpret them. Some maps were burdened with too much information, others were too vague, and Google’s map featured cryptic icons that had to be clicked to find the detailed information. Many people didn’t click.
• Informational websites must be failure-proof. Many participants went directly to PG&E’s website, assuming that the utility would be the best source of information. They reported that the website’s outage map and lookup function failed on the first day of the crisis.
• People must be educated in advance. Many of the participants in our test hadn’t realized they were living in areas that might lose power in a fire emergency. None of them were prepared for an outage lasting up to seven days.

Some other elements in the emergency response stood out because they worked well:

• Communicating broadly paid off. PG&E’s strategy of spreading information through as many outlets as possible created high levels of awareness very quickly. When we asked test participants if they were aware of the cuts, some said, “how could I not be?” That’s a sign of effective communication.
• Local news helped. Some local news sites (TV stations and newspapers) posted their own outage maps. Although these were not as detailed as the PG&E maps, their websites did not fail, and their information was much better than nothing.
• Text notifications worked fairly well. Many people reported receiving text and voice messages about the power cuts from PG&E and other government authorities (usually schools and local emergency services). The biggest frustration about the messages: they said outages “might” happen but didn’t give much certainty about when they would start or how long they would last.

Lessons for companies and organizations involved in public safety:

• Plan ahead. If you’re in an industry that might experience a public safety emergency, you need to stress-test your web infrastructure to ensure it absolutely will not fail in heavy traffic. 
• Invest in emergency text messaging. Most participants told us that getting texts is their preferred way to receive emergency information. The phone is always with them, and it’s easy for them to review text messages. (Several people said they also like to get emails, but reported that it’s easy to miss an important email because they are bombarded with so many marketing emails.) The more specific and up-to-date those text messages are, the more effective they will be at helping people prepare.
• Be as transparent as possible. Transparency builds trust. Explain why things are happening, not just what is going on. Be honest about uncertainties. For example, many people complained about the vagueness of the notifications they received (would power really be cut off, when would the cutoffs happen, etc). If the company had explained why the messages were vague, and when more information would be available, there would have been less frustration.

Methodology of the user tests

The tests were conducted during the power safety power cuts, on October 10. UserTesting panelists living in affected counties were asked a series of questions including:

• • Were you aware of the power cuts?
  • Were you notified that you might lose power?
  • Have you lost power?
  • How were you affected?

We then asked the participants to show us how they searched for information on the power cuts.

All responses were video-recorded. The test was run on twenty people, ten using desktop computers and ten using smartphones, for a total of about five hours of video. Representative clips were taken from the videos, and were edited for brevity and clarity. Tests like this are very accurate for detecting attitudes and usability problems, and understanding the “why” behind statistical surveys.