It was early morning and the clouds rolled in, turning the sky to darkness. The rain began pounding the building creating a rhythm that makes you want to close your eyes and sleep. And then I heard someone in Network Services yell out to the group that one of our credit unions was having trouble with their communications back to United Solutions. They were struggling to service their members. That escalated voice meant that if you weren’t already helping a credit union, that it was all hands on deck to investigate the issue.
At United Solutions, our strategy is that we are guilty until proven innocent. We weren’t sure why the credit union called in looking for assistance, but we immediately assumed that something was wrong with one of our systems. We have a battle tested process that we follow if connectivity is interrupted, and everyone begins investigating the problem. When the second credit union called in with a similar issue, the team immediately escalated to Senior Management that we had other credit unions calling in with connectivity issues.
Tony in Networking started a continuous capture on the PRTG Network Flow Analyzer, storing 15-minute captures on our network and running scripts looking for disconnects and reconnects. Sarah and Chris began checking network resources to make sure the servers and infrastructure were running efficiently and checking for anomalies that might point to a problem. As the third credit union called in, a Teams Chat was opened and everyone in Networking and the USC Management Team was participating in the conversation.
Everything looked fine at United Solutions as our systems appeared to be running as designed. One of the credit unions that called in said that two of their branches were fine, and just one was having trouble. The branches used two different Internet Service Providers (ISPs). That helped. We spoke to the other two credit unions that were experiencing problems, and they were both using that same ISP for their connection to United Solutions. Tony continued to investigate and correlate information from our communications equipment. He noticed that anything traversing a 170.55.xxx.xxx route had a ton of connects and disconnects. We called the ISP, a well-known national company, and let them know what we found. Two hours later the issue was resolved.
Guilty until proven innocent may not work in courts of law, at least not here in the United States. But in a network operations center (NOC), it is critical. You must assume that you are the problem and begin checking everything on your network while simultaneously communicating with third parties and your clients. Taking an attitude that you can’t be the problem and not taking responsibility immediately, lengthens the time to resolution and creates poor customer service.
In this case, the issue had nothing to do with United Solutions. The sporadic connectivity at a few of the credit unions was not caused by our data centers, but by “flapping” at one of the Internet Service Providers. We have a saying at United Solutions, “It’s not our fault, but it is our problem”. When our credit unions need help, it is all hands on deck and we take responsibility until proven otherwise. That’s good customer service!
**This article was first published in CU Today.**
