Long Days
Posted in Tech
First watch, 7 bells (11:59 pm)

I just got home from work—it's been a long day today. If you follow me on Twitter you already know. It started out okay, but around 2:30pm today our phone system shut down, right in the middle of all calls. After a quick sanity check that determined it didn't look like it was our problem, I called XO Communications, our service provider. At this point, it was about a quarter to three.

A trouble ticket was set up, and a technician called about 20 minutes later saying she was going to test our line. She called back just before 5pm and told me there are CRC and framing errors on our PRI. She said the only thing they could do was to set up a trouble ticket with Verizon and have their techs fix it. She also said it couldn't be done until some time after 8am tomorrow.

When I talked to my boss about it, he said it was unacceptable, which I agreed to, but didn't have much of an idea how to get it done. So I called my sales rep at XO to get some help and maybe accelerate the issue. I couldn't get through to anybody. Probably because it was just turning 5pm.

Anyway, finally I broke down and called the local XO technician's cell phone. Yes, I had it all along, but I don't abuse it. He is an awesome guy, and I really didn't want to call him unless I had exhausted my other options. He berated me for not calling him earlier! Anyway, he got the ball rolling with Verizon's support through his own channels. All I had to do was wait for their tech to call me and set up a time. Norm (our XO guy) said to call him back when the Verizon tech was going to be there so he could show up at the same time, just to make sure our problem got fixed. Like I said, he's a champ.

Anyway, an hour and a half go by without a call, and just as I'm about to call Norm back he calls me. He says "If you have a direct line to God now would be the time to use it." I was a bit worried. He said it has been one of the most difficult tickets he's ever had to work on. Finally, he pulls out the stops and calls Verizon's tech directly (just like I did by calling Norm). The Verizon tech had trouble getting his own office to release the ticket to him, so they had no idea what the hell was going on. Amazingly, this has not diminished my opinion of Verizon. But only because my opinion was already at rock-bottom.

Anyway, Norm gets the tech to stop by before his 10pm call elsewhere. He shows up just after Norm, they do their thing, and find that it's not a problem at the demarc (or d-mark, or whatever they call the place where all their lines come in to our building), but it's between the CO and us. That points them to a repeater that they've had trouble with before. Apparently it flooded and shorted out some electronics.

But before we tried to get that fixed, we thought we'd just jump on to an old T1 line that was provisioned for a company that used to be in our building. After they tracked that down from several different geographic locations, they had to give up because the line was came down from Hayden Lake where they couldn't get to 24/7. So we had to go back to plan A, fixing the flooded repeater. Verizon's emergency crew was dispatched to pump out the hole and replace the equipment.

At this point we'd done everything possible, and had to leave it in the hands of the emergency crew. They promised to get us up and running before we open the doors tomorrow morning at 6am. I really hope it works out, but in the meantime have worked out a new plan B with John, my cow-orker. We can forward our primary DID (where all our 800 numbers are routed) to a new number, such as a cell phone. By putting a voicemail message explaining the outing, we can at least let people know what's going on instead of them just getting a busy signal.

Leave a Comment »

Leave a Reply