When disaster strikes, one of the most valuable tools you have at your disposal is your phone. Yes, food and shelter are more important and more immediate needs, but your phone enables you to find out what’s going on, get help, and let loved ones know that you are ok. You can have the best smartphone in the world, but the network it runs on can also be vulnerable in the face of disaster. Whether it’s floods, explosions, or hurricanes, cell towers and the power lines that make them work can also be affected. So it’s critical that your carrier can keep you connected when disaster strikes.
AT&T takes network reliability very seriously, and nowhere is this more evident than in AT&T’s handling of emergencies and disasters — natural or otherwise. AT&T flew me down to New Orleans, Louisiana to show off one of its disaster response installations and to talk about what a disaster looks like through the lens of AT&T. During the tour, we talked about how the company plans for emergencies and has teams of full-time staff and volunteers at the ready.
Disaster preparation
The first part of disaster recovery is being ready to handle anything that comes your way. To that end, AT&T has twenty locations placed around the country that are fully stocked with any equipment necessary to recover and restore communications in the event of an emergency. AT&T can have gear in place basically anywhere around the country in about six hours.
But AT&T needs more than just gear. They have a limited full-time staff of people who head up disaster recovery operations, including the maintenance of equipment at these twenty sites. On top of that, they have a volunteer force of around 400 employees including managers, technicians, and engineers that are ready to go at the drop of a hat. These volunteers have so-called “go bags” ready at all times with three days’ worth of clothes, equipment, and provisions (including food and water) ready so they can be off to a disaster site, sometimes in as little as 30 minutes.
Additionally, AT&T’s staging areas have everything they need on hand to get to a disaster site, start repairs, and maintain its network in the meantime. This includes trucks, forklifts, generators, portable 5G stations, fuel, tools, and more. I asked AT&T what were some of the provisions that they didn’t initially think of, but turned out to be needed during an emergency. The first was fuel for AT&T employees’ cars. Gasoline pumps can break down during an emergency, so now AT&T regularly keeps a supply of unleaded gasoline on hand so employees can get to and from emergency locations.
Another consideration AT&T learned about was the need for cash. In an emergency, local economies become especially reliant on cash. AT&T has learned to keep a supply of cash on hand during emergencies, and no, they wouldn’t tell me where they kept it.
It’s one thing to know where to store your gear, but it’s another thing to have a place to keep your workers. To that end, AT&T has a series of living quarters on trailers that can house up to 10 workers (five per side) in separate rooms with their own lights, power outlets, and AC units. These can roll up onto a scene and only require a generator to be hooked up to be fully livable.
Reestablishing networks and FirstNet
When emergencies take out AT&T’s networks, there’s really only one way to fix it fast, and that’s to roll in a temporary network on wheels and set that up as fast as possible. Once that’s done, AT&T can begin the work of determining the cause of the network outage, confident that communications are restored. Communications are key not only for customers, but also for other first responders that are on the scene including law enforcement, search and rescue, civil engineers, and more.
To that end, AT&T has two different kinds of portable towers at its disposal. The first is known as a Cell on Wheels (COW). This is a (relatively) small, 500 lb unit on wheels that has a tow hitch. A truck can carry a flatbed or trailer with a number of these and drop them off, or they can be towed behind smaller trucks to get where they need to go. They can be set up in 15 to 30 minutes and can extend an antenna to provide coverage for up to 50 active users within a 2-mile circumference, as long as they have a clear view of the southern sky. The units connect to geosynchronous satellites to provide connectivity and network bandwidth. The COW is small enough to fit into most elevators and doorways so it can be set up quickly on the top level of a parking garage for greater coverage.
If more coverage is needed, AT&T brings in a Satellite Cell on Light Truck or SatCOLT. This is a truck that can drive out to wherever it’s needed, extend self-leveling legs, and raise a mast 60 feet in the air. Inside the truck is a small office with all the equipment needed to set up and run a cellular network, along with a small office, battery backup, and HVAC system. The SatCOLT has a built-in generator, but it typically runs off a separate generator that AT&T sets up with the truck. If the external generator dies, the built-in generator takes over, and if that dies, the battery backups kick in. Meanwhile, the network operations center gets alarms indicating what went down so teams can be sent to fix things before everything goes down. AT&T has a fleet of approximately 80 SatCOLTs around the country.
These portable towers help support regular customers, but they give priority to first responders using AT&T’s FirstNet network. FirstNet is a network co-developed by AT&T and the FirstNet Authority to build a nationwide cellular network for first responders. AT&T manages that network. One interesting thing about FirstNet and this hardware is that FirstNet devices have what’s called a “black SIM” which gives it automatic priority in the FirstNet network above a common user. This is done automatically without any need for whitelisting or other techniques that competing first responder systems use.
Beyond FirstNet, AT&T also has non-FirstNet equipment it can set up which works for all traffic normally. It acts and behaves just like a normal cell tower, and it is completely transparent for the user experience. As far as anyone is concerned, they’re just making a phone call or surfing the internet. These remain in place until the actual infrastructure is repaired and reactivated, and the users are none the wiser.
Land or sea
When things get really bad, especially in a hurricane-prone environment, sometimes you need to be able to take to the waves. Flooding can be just as disastrous to a network as high winds. In times like those, boats and watercraft can be hard to come by, so AT&T owns and can send out three Hydratrek amphibious vehicles. These are tracked vehicles that drive up to 15 mph on land or 3 mph in the water. They’re basically floating flatbed trucks that can haul up to 2,000 pounds of personnel or gear, depending on the need.
This vehicle is the embodiment of AT&T’s philosophy when it comes to disaster relief. AT&T’s first tenet is to bring everything you need yourself so you don’t tax local resources. In an emergency, everyone is going to need basic kinds of equipment like forklifts, trucks, diesel fuel, food, water, watercraft, and more. AT&T brings it all along to keep their own services running and so as not to tax resources needed by other groups. Therefore if AT&T can’t be sure about what kind of terrain it may need to traverse, they bring three vehicles that can traverse basically anything. Additionally, AT&T’s trucks are equipped with chainsaws, tools, chains, lights, and everything needed to clear roads and build up infrastructure if necessary.
Overall, all this equipment and technology points to how fundamental communications are in a disaster. People need to know where to go, where help is needed, where it’s safe, and where food, water, and shelter can be found. As a communications company, AT&T takes that very seriously, and to be fair, T-Mobile and Verizon also have disaster recovery plans — this tour just happened to be at AT&T’s facility.
When the worst things happen, it’s good to know that there are teams and volunteers standing ready to dive into the fray at a moment’s notice to help customers and first responders get online so they can get things back to as close to normal as is possible as quickly as possible.