On Outages and Early Mornings
Posted in General
First dog watch, 3 bells (5:34 pm)

This morning I went to the office early to make sure the new website rollover went smoothly. I got there just after 7am to find the phones down because our T1 PRI was offline. Fortunately, most of the right steps had already been taken to resolve the problem and I just had to make a few phone calls. Well, after the heat from that died down, the website switch was green-lit. It took me about 12 seconds to change the Apache config.

Most everything else was solving small problems. A week or so ago I wrote a PHP script to handle importing data from an online database to an internal server, but due to the complicated nature of the data, it was difficult to test. I took a long time to set up an environment to test against, and even then it still wasn't really conclusive. But it looked good, so I deployed it and it runs fine. The only thing that worries me is that it worked without any bugs at all. It always makes me nervous when something just works, it makes me wonder what nasty bug I haven't found. Anyway, I had to fix an issue, and the entire fix was implemented by adding the number '6' to the file at a certain spot. One-byte change. It's great to crank something like that out once in a while, but I'd rather have something I know had bugs and was fixed…