October 30, 2008 Leave a comment
Today was one of those days. The network was down when I got to work, so even before my first cup of coffee I was fielding “I can’t..” and “It doesn’t…” We’ve been having some issues with one of our older 3com switches and that was my first thought. I did a reboot of that switch but that didn’t clear things up. It did bring back internet but my fileserver was still offline (ping-able) but nobody could connect. I did a reboot of that server and while I was waiting for it to come back up I had a look at the last vulnerability scan for our AD/DNS server. This scan is done by eEye software’s Blink which is built on the highly regarded Retina vulnerability scanner. I hadn’t checked the scan results for a while (bad me, I know… but at least I AM scanning. Are you??) And the thing that immediately caught my attention was a HUGE number of open UDP ports. (in the thousands!!) The port numbers range from 49157 to 65529.
Immediately I’m thinking I’ve either been hacked or someone has set up some P2P on our network. I was pretty sure it wasn’t someone internal so I immediately went into intrusion event management mode. Checking logs, current firewall traffic, A/V scans and then a bunch of malware scans. Blink is has a good set of tools for this and it was coming up empty. I went to my next line of defense which is good old Spybot. Again the system came up clean. I also used a tool from eSet (makers of NOD32 A/V) called SysInspector. This is a useful utility I found on a sans.org diary posting. Nothing out of the ordinary there either. Lastly, I did a scan with RootKit Revealer from sysinternals err Microsoft. Again, nothing.
I should point out that by this time everything is working fine for all my users. You might be thinking that by this time, I’m feeling a bit better about things. Nope. In fact I’m even more worried as I’ve got a ton of open UDP ports and I can’t find a reason why. My next move is to create a custom rule for the Blink software firewall, blocking the open UDP ports and log the results. All of a sudden, the Blink service goes to 100% cpu and I’m getting a -lot- of denied log entries for all of these various ports. Each of these ports seems to be trying to contact a different external IP. Now I’m thinking I’m really screwed. Someone is operating some kind of server on my -clean- system!
But then… I get a call. “I just lost my internet”, then another. Then -I- can’t browse to some sites. Hmmm. DNS? I did an ipconfig /flushdns /registerdns on my workstation and now I have no name resolution. Ok. It’s DNS related for sure. I go back to my AD/DNS server and remove my newly created rule repeat the ipconfig and viola… the internets are back.
So, I did a google search for UDP 49157 to 65529 and didn’t really turn up anything. But I did finally come up with a mention of them being part of the dynamic or emphemeral port range. Once I found that out I found the wikipedia entry which gave my the entire range of these ports (49152–65535). With that and a quick search, I was finally led to an article on TechTarget which has an explanation.
“This security update also introduces a new default for DNS port settings for Windows Server 2000 and Windows Server 2003 — dynamic default socket port ranges have changed from 1025 through 5000, to the new range of 49152 through 65535. We encourage you to review the firewall settings in your environment to ensure that traffic between servers in the dynamic port range of 49152 through 65535 is allowed. Windows Vista and Windows Server 2008 already have the default port range of 49152 to 65535. For additional information, please review the MS08-037 bulletin.”
Lightbulb moment. All those open ports are supposed to be there. They are the ports which are used by the new DNS patch to avoid the DNS cache vulnerability of July 2008. Sigh. I probably could have stopped after rebooting the fileserver (oh, about 6 hours ago)
I don’t feel that it was a wasted day for a couple of reasons. Firstly, like most other short handed IT folks, I never make the time to do incident response simulations. This was a good -simulation-. Secondly, I’m pretty confident that my various levels of security, patching and updating are doing their jobs. Lastly, I’m happy to report a clean bill of health for my server room.
It’s too late for that cup of coffee, but it’s just the right time for a drink,