These people are not hackers, but tiny-minded people trying to up search engine results and send out spam. If you want to know what a real hacker is, read 2600 Magazine.

Anyway, I deleted their exploit, then realized I was missing an opportunity, and replaced the page with this:

Now if they ever come from an un-banned IP address (yeah, I shut them out, and the rest of their ISP's network, too!), that's all they're going to get…

Update: Well it seems like this was a bit premature. It appeared that there was a DHCP issue, but the real problem turned out to be a Cisco 2960 switch behind the GS108 that had the specific port configured to "Desktop" mode. A change to "Switch" mode fixed the problem, so if you have an issue like this, check your other hardware!

Work has been busy—very busy. I'm behind on projects because other things come up. My backup system at the office died and I've got a lot of new equipment in to replace the old system. Having chosen to upgrade instead of replace hardware (it was five years old), I had several other peripheral systems that needed upgrading as well. Because my new LTO-5 tape drive has a SAS connection, I had to get a new server that supported SAS. And because I needed a new server, I had to move all the data storage to the new server, but the old storage was Ultra320 SCSI. So I had to buy a new storage vault. It uses iSCSI, which is some very cool tech: it's a network block protocol, and connects to my server via gigabit CAT5e. And since I was replacing just about everything, it was time to upgrade my backup software, too.

I've only today got things working and have high confidence that I will have a full, clean backup tonight.

Hopefully this means I can concentrate on getting my other work finished now.

On a side note, Lorien and I have memberships at the Cd'A Kroc Center and I'm committing myself to work out there every weeknight. You can catch me there starting around 8pm. It's a really nice place, and we're excited to have a plan to get fit together.

I soon found out that some of my coworkers see the same problem, but only those that have Time Warner cable at home—but not all of those that have Time Warner see the long delay.

Many theories were thrown out about what was happening: a DNS issue, a firewall, some kind of faulty caching proxy server, etc., most of which were ruled out because we didn't see the same problem from all locations and that I could reproduce the problem on two separate servers. Watching the webserver's logfile showed no entries for the 30 seconds I waited and then all of a sudden the log was populated and my page loaded quickly.

Examining the differences between the clients with and without the problem, I noticed that those that behaved as expected had working reverse-DNS entries, while those that had trouble returned an error when I queried for their reverse DNS address.

So as a test, I disabled reverse-DNS on my SSH server at my office and BAM—the delay was gone!

Continuing the experiment, I added a record for my home IP address on my server's /etc/hosts file. Just like that, I was loading pages at full speed with no delays.

When I query for a reverse-DNS resolution of my home IP address, Time Warner's server returns an error. Contacting them to have them fix their broken DNS server has been an awful experience, starting with "Let's reboot your router first."

Since I expect this to never be resolved, and since I run my own DNS servers anyway, I started adding in-addr.arpa zones for the broken ranges of DHCP addresses. While this will work for me, it's a hack at best and I would rather get the issue fixed. Unfortunately it's out of my hands.

So Time Warner, surprise me. Fix it up, or un-configure the range of addresses even! Don't just leave it broken.

I have seen nothing else like this guide on the Internet, probably because Cisco consultants jealously guard their secrets to justify their extremely high hourly rates. Time for a change…

So my first though was to fdisk /mbr the thing, but fdisk is no longer part of Windows. That gave me a sad. Fortunately, there's a new alternative: bootsect /nt60 c: /mbr

Running that and rebooting took me right back to the installed operating system, so saving itself from a pending reformat.

PS. The default IP address of a Cisco 881W router is 10.10.10.1 with a netmask of 255.255.255.248 (or /29), and the usable range of IP addresses for that setup is 10.10.10.1 to 10.10.10.6.

I'm going to cash in some chips with some business associates and get some help finding the issue, it's a bit over my head at this point, and that doesn't happen to me very often.

It's not latency, because ping times are nominal. Yet traceroute results may be delayed for 30+ seconds while I wait for the second hop (first is my home router) to return results, despite having 10ms or less latency. It's like the connection itself is being held up somewhere, and seems to be happening mostly on a hand-off from Time Warner to XO Communications' network.

I'll post updates when I get closer to finding the issue.

In any case, after triage I was doing some system checks and I found I'm missing the /var/log/messages file. Weird. So I send a SIGHUP to the syslog daemon and it does nothing.

So I check the /sbin directory and the klogd binary was there, but the syslogd binary wasn't! I had to reinstall syslog-ng. It's hard to track down a problem like this when your logfiles aren't there…