AT&T places high value on keeping its 145 million wireless and 16 million broadband users connected at all times. A short 20-minute service break can lead to significant costs because the company issues bill credits during any loss of access.

Most interruptions stay limited in scope and rarely draw widespread notice, yet occasional events impact millions of users. In February 2024 a large wireless failure at AT&T prevented more than 92 million voice calls and over 25,000 attempted 911 connections, according to a US Federal Communications Commission report. Rival carriers Verizon and T-Mobile have faced similar large-scale problems in recent years.

To handle outages more efficiently and communicate better with customers, AT&T moved to consolidate its approach for spotting network problems.

"We must detect every network interruption right away and resolve it, preferably before customers notice any impact," said Andy Markus, AT&T chief data and AI officer.

Starting in 2017 the company developed its end-to-end incident management system, known as EEIM, using platforms from MongoDB and Snowflake. The system began with conventional machine learning and later incorporated generative and agentic AI features that expanded its capabilities, Markus noted. It now supports fixes for both individual customers and small businesses.

The EEIM combines AI and machine-learning functions to determine the source of a problem, prioritize solutions that can be handled remotely or require on-site staff, and notify customers ahead of time when service is affected, Markus explained.

The Tech

Network problems arise from many sources such as severe weather, equipment faults, software errors, security breaches, and physical damage. When users experience issues with phones or internet service they often contact support centers in large numbers.

Markus noted that in early 2017 AT&T began building a more responsive outage system. A cross-functional group from IT, data, and AI teams contributed ideas to create the EEIM while adding new AI tools to daily processes.

Input also came from field technicians who maintain the network and from network staff who track traffic and restore service quickly, Markus said.

AT&T used the EEIM to reorganize 10 petabytes of information, equal to roughly 5,000 billion printed pages. The data included network logs, alarms, dispatch tickets, and outage records. Consolidation allowed faster issue detection, prediction of future problems, and better coordination of responses.

MongoDB supplied a flexible document database that supports growth by adding capacity rather than redesigning the system, said Ben Cefalo, MongoDB chief product officer. This approach, called sharding, spreads data across machines to manage large volumes effectively.

The EEIM also uses Microsoft Azure for cloud services, Databricks for analytics, and Snowflake for incident reporting, Markus said.

An AT&T spokesperson confirmed the EEIM began supporting broadband fiber in the first quarter of 2018 and added digital subscriber line coverage the next year.

In June 2018 the company introduced the Atlas application for field technicians. It blends machine learning and AI to identify outage causes and suggest repair steps.

AT&T had already adopted traditional AI methods before Markus joined in July 2020, after which broader AI use expanded. He reported that 100,000 employees now access generative AI tools, processing over 27 billion tokens daily while fine-tuning smaller models to control expenses.

By the first quarter of 2021 the proactive customer alert system powered by the EEIM went live. "It reduces customer stress and annoyance," Markus said.

The Outcome

Generative AI capabilities joined the EEIM in the first quarter of 2022, allowing the platform to recognize outage causes by referencing past events, Markus said.

In the first quarter of 2025 AI agents were integrated. They can collect outage details from customers, initiate resolutions, and share information with technicians when field visits are required.

AT&T has created more than 30 AI models that forecast issues tied to configuration errors, weather, or equipment failures, supporting earlier intervention.

"The system helps us examine signals and patterns to spot current or upcoming problems," Markus said.

Overall the AI-powered EEIM has avoided 3.1 million unnecessary field visits and cut customer downtime by more than 12 million hours in the past year, Markus stated.

AI Chain Reaction
AI
automation
More
Generative AI
Artificial Intelligence
Infrastructure