Today I fired off a broadside blast against some annoying script kiddies that have been defacing a vBulletin site I have the unfortunate misery to manage. I finally tracked their crack down to a PHP file that was uploaded under a previously vulnerable version. I had no idea it was there, but these evil files kept showing up despite my keeping the software updated.
These people are not hackers, but tiny-minded people trying to up search engine results and send out spam. If you want to know what a real hacker is, read 2600 Magazine.
Anyway, I deleted their exploit, then realized I was missing an opportunity, and replaced the page with this:
Now if they ever come from an un-banned IP address (yeah, I shut them out, and the rest of their ISP's network, too!), that's all they're going to get…
My boss bought a Netgear ProSafe GS108 gigabit network switch to use around the house, but had a lot of trouble with it. As it turns out, the device has a firmware bug that does not allow DHCP packets! I'm not sure about other broadcast protocols, but if you need DHCP, don't use this switch.
Update: Well it seems like this was a bit premature. It appeared that there was a DHCP issue, but the real problem turned out to be a Cisco 2960 switch behind the GS108 that had the specific port configured to "Desktop" mode. A change to "Switch" mode fixed the problem, so if you have an issue like this, check your other hardware!
I've been blogtarded lately and haven't written anything for quite a while. I guess Twitter has filled what little output needs I've had lately.
Work has been busy—very busy. I'm behind on projects because other things come up. My backup system at the office died and I've got a lot of new equipment in to replace the old system. Having chosen to upgrade instead of replace hardware (it was five years old), I had several other peripheral systems that needed upgrading as well. Because my new LTO-5 tape drive has a SAS connection, I had to get a new server that supported SAS. And because I needed a new server, I had to move all the data storage to the new server, but the old storage was Ultra320 SCSI. So I had to buy a new storage vault. It uses iSCSI, which is some very cool tech: it's a network block protocol, and connects to my server via gigabit CAT5e. And since I was replacing just about everything, it was time to upgrade my backup software, too.
I've only today got things working and have high confidence that I will have a full, clean backup tonight.
Hopefully this means I can concentrate on getting my other work finished now.
On a side note, Lorien and I have memberships at the Cd'A Kroc Center and I'm committing myself to work out there every weeknight. You can catch me there starting around 8pm. It's a really nice place, and we're excited to have a plan to get fit together.
For the past few weeks I've been working on a strange issue from my Time Warner cable connection at home. There were two symptoms: a long delay of roughly 30 seconds when trying to open a web page on my company's webserver, and a similar delay when connecting to my office via secure shell (ssh).
I soon found out that some of my coworkers see the same problem, but only those that have Time Warner cable at home—but not all of those that have Time Warner see the long delay.
Many theories were thrown out about what was happening: a DNS issue, a firewall, some kind of faulty caching proxy server, etc., most of which were ruled out because we didn't see the same problem from all locations and that I could reproduce the problem on two separate servers. Watching the webserver's logfile showed no entries for the 30 seconds I waited and then all of a sudden the log was populated and my page loaded quickly.
Examining the differences between the clients with and without the problem, I noticed that those that behaved as expected had working reverse-DNS entries, while those that had trouble returned an error when I queried for their reverse DNS address.
So as a test, I disabled reverse-DNS on my SSH server at my office and BAM—the delay was gone!
Continuing the experiment, I added a record for my home IP address on my server's /etc/hosts file. Just like that, I was loading pages at full speed with no delays.
When I query for a reverse-DNS resolution of my home IP address, Time Warner's server returns an error. Contacting them to have them fix their broken DNS server has been an awful experience, starting with "Let's reboot your router first."
Since I expect this to never be resolved, and since I run my own DNS servers anyway, I started adding in-addr.arpa zones for the broken ranges of DHCP addresses. While this will work for me, it's a hack at best and I would rather get the issue fixed. Unfortunately it's out of my hands.
So Time Warner, surprise me. Fix it up, or un-configure the range of addresses even! Don't just leave it broken.
If you follow me on Twitter, you know that I've been learning the Dvorak keyboard layout over the last couple of weeks. Yesterday I switched my main work computer over to use Dvorak and although I'm typing slow now, I'm getting better. I also have some empathy for people that can't type, as my typing speed is now quite slow.
I just wrote up a long page about how I configured a Cisco 881W router to fetch an external DHCP address from my Internet provider, serve internal DHCP to my clients, and set up a wireless network with WPA encryption.
I have seen nothing else like this guide on the Internet, probably because Cisco consultants jealously guard their secrets to justify their extremely high hourly rates. Time for a change…
I almost had to reformat a Windows 7 machine today that wouldn't recognize the primary drive or the copy of Windows installed on it. Running the repair option from the DVD saw the secondary drive and gave me no option to fix the master boot record. Opening a command prompt (again, using the Repair section of the DVD) defaulted to the D: drive, but allowed me to change to C: and view files just fine.
So my first though was to fdisk /mbr the thing, but fdisk is no longer part of Windows. That gave me a sad. Fortunately, there's a new alternative: bootsect /nt60 c: /mbr
Running that and rebooting took me right back to the installed operating system, so saving itself from a pending reformat.
This article about connecting to a Cisco router with minicom just helped me solve a nasty problem I've been having.
PS. The default IP address of a Cisco 881W router is 10.10.10.1 with a netmask of 255.255.255.248 (or /29), and the usable range of IP addresses for that setup is 10.10.10.1 to 10.10.10.6.
I'm trying to track down a very strange issue that seems to be geographically related, but not completely so. Basically, certain routes to some servers have long connection delays, sometimes 30 seconds or more, before any data are transferred. I can reproduce it at home some of the time. Other times, the problem is just gone. Several other people at work see the same issue, but others with the same service don't. I told you it was strange&8230;
I'm going to cash in some chips with some business associates and get some help finding the issue, it's a bit over my head at this point, and that doesn't happen to me very often.
It's not latency, because ping times are nominal. Yet traceroute results may be delayed for 30+ seconds while I wait for the second hop (first is my home router) to return results, despite having 10ms or less latency. It's like the connection itself is being held up somewhere, and seems to be happening mostly on a hand-off from Time Warner to XO Communications' network.
I'll post updates when I get closer to finding the issue.
One of my servers went down today. It runs Red Hat Enterprise Linux 4. I don't know what happened, none of the partitions filled up or anything, it just "freaked out." Perhaps something went bad somewhere, and in its defense, it'd been running for 352 straight days.
In any case, after triage I was doing some system checks and I found I'm missing the /var/log/messages file. Weird. So I send a SIGHUP to the syslog daemon and it does nothing.
So I check the /sbin directory and the klogd binary was there, but the syslogd binary wasn't! I had to reinstall syslog-ng. It's hard to track down a problem like this when your logfiles aren't there…
Today I received the Drobo FS I ordered last Friday, along with three 2Tb drives to make it happy.
First off, can I say that in my years of IT work, this is by far the easiest installation I've ever had. All I did was slide the drives in and plug in the network and power and turned it on. Then I walked away. Sheer brilliance.
I installed their software, the Drobo Dashboard, so I could configure the device (from a Windows machine). A few minutes after power-up, there it was in the window showing drives ready to go:
The only complaint I have is interoperability with backup software (I use Symantec's BackupExec), which appears to be available only with the Drobo Pro and Drobo Elite models. Don't get me wrong, I'm sure I can pull data off via the share, but a backup client does a better job of streaming data out than Windows SMB.
Overall, I'm excited to have an appliance to store our data without having to maintain an operating system and all its related hardware. Depending on how things go, we may be switching more of our storage over to Drobo later.
Now that my MUD (which I've recently settled on the name ForeverMUD) is getting more stable, I thought I'd run some profiling tools on it. Valgrind is a very cool utility made to do this very thing.
When I first started developing, I wrote in a loop timer that keeps track of the longest processing time as well as the average loop time. After the introduction of a large (16,200 room) world map, the loop times jumped from 10-30 µsec to over 190,000 µsec—no bueno.
So last night I ran Valgrind and used kcachegrind to view the results, which showed me this:
Put simply, most of my time is spent manipulating mutexes. I'm going to have to rethink my design, because that's way too much time…
Wow, I heard the Microsoft Kin's been axed already. I just started seeing commercials for it on Hulu the other day. It's only been available for two months I think.
I guess if you have a gajillion dollars you can make stuff and never sell it as often as you want. Still, it seems like a waste to me…
Today my boss came in to my office and asked if I could find out the output voltage and impedance for his laptop's sound card.
Hardware specs tend to not go that in-depth. Finding the manufacturer of the sound hardware may work, but I ran in to several dead ends trying this.
In the end, I found a website dedicated to using your sound hardware as a cheap oscilloscope. Turns out, they've done the research and listed averages, which ended up being good enough for my boss.
The answer? Typical output voltage for a computer line-out jack is 1.5-2.0v. Impedance tends to be between 30 and 400 ohms. Just in case you needed to know.
I fixed a small but annoying (to me, anyway) bug in my blog theme just now. If you were to view my custom theme in a very wide window (greater that 1400 pixels or so) the main body text would start to "underlap" the right-hand sidebar. By imposing a maximum-width limit to the content area I fixed the issue to my satisfaction.