Brazos Outages Likely Caused By Software

Young County residents and customers across North Texas experienced intermittent internet outages beginning Sunday evening, with brief service interruptions continuing into last week. Brazos Communications reported disruptions, though service has largely stabilized, according to Brazos Communications CEO Kyle Hinson.

“Brazos Communications has and is continuing to spend millions of dollars in investment and hired the best employees to provide broadband to the area we serve,” Mr. Hinson said. “Starting in 2014 and continuing today to install fiber for the most future-proof and fastest speeds available for our customers to be able to access the vast amount of information that the internet has to offer. The commitment to service for our customer is number one for the board and our employees.”

As for the outage, the longest disruption occurred Sunday night, lasting several hours. Since then, outages have been shorter — “five to fifteen minutes per day over the next four days,” Mr. Hinson said in an interview Wednesday.

The interruptions, he said, were tied to software programming conflicts, not equipment failure or malicious activity. By Friday afternoon, there had been no disruptions for over 24 hours, he said.

“There is no hardware that has shut down, and nobody hacked into us,” Mr. Hinson said. “This was an unplanned software issue. Sometimes referred to as a Software Data Overload, where one system may overload another and shut down the server, then corrections have to be made.”

Mr. Hinson emphasized that the cause is still under review. Internet service relies on many layers of software — spanning customer equipment, network infrastructure, and the cloud — and when one layer fails to mesh with another, disruptions can occur.

“Our internet technicians have been working nonstop to correct this outage - a few staying up over 30-plus hours straight. The software programmers spent hours looking at their logs that record everything. The software is multilayered, and that’s where the conflicts can come from,” Mr. Hinson said. “If you find an error when you update it, you have to go in and tweak it. These adjustments can cause a short outage also.”

Mr. Hinson noted that software updates can be performed weekly or monthly by outside programmers, typically in the late evenings. While the timing is meant to minimize disruption, problems can surface the next day when network traffic spikes.

In a statement posted to customers, Brazos Communications acknowledged the interruptions and reassured subscribers that steps are being taken to improve reliability. “Brazos is investing in our network and upgrading it to provide the best service experience for our customers,” the statement read. “As an internet technician once told me, it’s like changing a tire on a moving bicycle. We have become a society that is online 24/7, so there is never a good time to do updates.”

Brazos Communications said it is continuing to monitor the network and make fine-tuning adjustments. Mr. Hinson urged customers still experiencing issues to call the support line at 800687-3222.