When the lights went out March 4, then again March 17 and for a third time March 25, students and faculty across UTD’s academic core found themselves in a familiar position: staring at dark computer screens, locked out of labs and refreshing social media for answers.
In a virtual town hall held March 31 — the second such attempt at a town hall after the March 25 one was canceled because of the power outage — Rafael Martín, vice president and chief of operations, alongside other operations leadership offered the most detailed account yet of what went wrong and what it will take to make sure these outages don’t happen again.
The short answer is that a contractor drilled a hole through the university’s central underground cable network. A network that, after two decades in use, is already aging and needs urgent change.
The long answer involves millions of dollars, months of construction and a fundamental reevaluation of how UTD generates and distributes power to a campus that has outgrown its original infrastructure.
“We know that these types of disruptions have real costs to members of our community,” Martín said. “We are very sorry for those losses.”
We know that these types of disruptions have real costs to members of our community. We are very sorry for those losses.
— Rafael Martín, vice president and chief of operations
These sorely needed and costly upgrades do not come as a surprise to the UTD community. Last year, Michelle Rinehart, academic safety program manager with the Office of Institutional Risk and Safety, said during a presidential search town hall that UTD was behind on millions of dollars’ worth of back maintenance. Rinehart said she feared it could become a dangerous bubble.
“[We need to] put an effort into this balance of trying to get our name out there, trying to grow, trying to do all these exciting things and making sure we don’t crumble,” Rinehart said. “I know that it’s not glamorous, but we’re reaching a little bit of a bubble and I’m afraid we’re going to pop.”
And it did pop. Three times.
Three outages, three causes
UTD has four total feeder systems, essentially underground lines, that bring power to the entire campus. The first outage was traced back to a failure in the B-phase cable conductor of the M1 feeder, one of the two lines that powers the entire academic core. The M3 and M4 feeders provide power to facilities on the east side of campus and housing on the west side, respectively. To fix the failure in M1, crews shifted the load typically shared between the M1 and M2 lines entirely onto M2.
This arrangement worked until March 25, when a separate issue — a planned repair to a chiller in the satellite utility plant — caused the campus’ building control systems to demand more cooling power. The M2 feeder, now carrying the full load of two lines, was overloaded and tripped its circuit breaker.
“We are doing it the old-fashioned way now,” Kelly Kinnard, assistant vice president for facilities services and operations, said. “Keeping everything balanced by hand to keep our load well below the trip settings. We are stable at this time.”
Between those two outages, however, came a longer and more frustrating one for administrators. Beginning March 17 and lasting through March 21, power to the academic core was knocked out by a contractor working for Atmos Energy, a natural gas supplier for the school. Martín said the contractor was replacing a gas line and failed to avoid the underground duct which carries both M1 and M2 through campus.
“They drilled through several bundles of cables,” Martín said. “Three of the four bundles [in M1] were completely drilled through. The fourth had damage to a single cable as well.”
Martín said the contractor did not immediately notify the university.
“It’s not unusual when a subcontractor causes damage for them to fail to notify us,” Jeremy Head, associate vice president for facilities management, said. “Had it been done in this case, Facilities Management could have responded literally days sooner.”
The university is now pursuing “claims” against Atmos and its contractor for the cost of repairs and for damages sustained by faculty and researchers.
Aging infrastructure’s $2.5 million fix
M1 and M2 were installed in 2007 and 2008, respectively, and networks like them typically have a rough lifespan of 25 to 30 years. Years of splices, additions and routing elbows have taken their toll.
“We didn’t even make 20 years,” Kinnard said.
The solution, officials said, is to replace both feeders entirely. The work is expected to cost approximately $2.5 million and will require two planned four to six-hour shutdowns, currently planned to take place during the weekends, to energize temporary bypass feeders and then reconnect the new cables.
“We will provide campus with as much advance warning of those shutdowns as possible,” Martín said.
Looking toward the future, the university is launching an ”N+1 study” to help ensure critical facilities are less impacted by outages. The study will evaluate what it would take to add backup generators or redundant power paths to buildings that currently have neither.
“Uninterruptible power supplies come with some pretty high costs,” Kinnard said. “If individual labs would like to put a UPS for their lab, we would ask that they contact us. But we are absolutely in favor of it.”
Communication breakdowns
Throughout the three outages and during the town hall itself, students and faculty complained about slow and inconsistent updates. Faculty submitted multiple questions during the town hall asking why there was no text alert during the March 17 outage.
Martín said the decision was ultimately based on the timing of the outage — it was the middle of spring break, when few students were on campus — and a policy of reserving text alerts for “true emergencies.”
“We felt that email and social media notifications were the appropriate way to communicate,” Martín said. “I understand that some employees may not have gotten those messages and come to work on the 18th and realized then that the power was out. For that, I apologize.”
Going forward, facilities leadership pledged to provide more timely and transparent information. For planned maintenance, notices will continue to go out weeks in advance. For unplanned outages, the goal is to have information out within the first hour.
Our goal is to provide accurate information about when we expect to be able to restore power, or if we don’t know, a timeline for when additional information will be available.
— Rafael Martín, vice president and chief of operations
“Our goal is to provide accurate information about when we expect to be able to restore power, or if we don’t know, a timeline for when additional information will be available,” Martín said.
The limits of backup power
One recurring question from research faculty was, “Why don’t buildings have generators to keep labs running during these outages?”
The answer, officials said, was that most campus generators are designed only for “emergency backup for fire and life safety.” They can supply enough power to light exits and run elevators, but not enough to keep negative 80 degree Fahrenheit freezers cold or fume hoods operational.
The Bioengineering and Sciences Building and the Natural Science and Engineering Research Lab are the exceptions to this rule. They have generators capable of backing up 60 to 70% of the buildings’ electrical load. That capacity, Kinnard said, was “more of a beneficial happenstance than on purpose.”
For now, researchers who sustained losses during the outages — like damaged equipment, lost samples or ruined experiments — are being asked to document their damages for the university to submit as part of their claims against Atmos and its contractor for reimbursement.
“We will be reaching out to administrators across the university later this week to start soliciting that information,” Martín said.
What comes next
Martín said the immediate priority is replacing the M1 and M2 feeders. Officials expect to identify a contractor in the coming weeks and will announce timelines for the upcoming planned shutdowns as soon as those are available.
Beyond that, the N+1 study will shape longer-term investments in redundancy and resilience for the campus grid. However, officials said that full-building backup power for every facility remains financially out of reach.
“You would typically find that in health care environments, where you cannot lose power for any extended period,” Martín said. “That is an expensive project.”
For now, the campus remains on what Kinnard described as “stable” footing: the M2 feeder carrying the academic core’s load, additional above-ground support, systems being monitored at least four times per day and technicians manually balancing the system to avoid another circuit break.
“The data is supporting that what we’re doing is effective,” Kinnard said.