How Secure is the City’s Computer Network?
Two weeks ago, on Tuesday May 15, 2006, my website was attacked. I did a post the next day but did not share any details on the source. Well, it was from the City of St. Louis. Not within the city limits but from the government of the City of St. Louis.
For about an hour and a half a server(s) asked for my main page at a rate of twenty times per second. At the time I characterized it as a deliberate denial of service attack.
I know a bit more now so let me share what I’ve been told. First, depending upon who you talk to you get a different answer — typical with technology issues. The chart at the right shows information on visits to my site all in mostly cryptic IP address. The top one, however, has been confirmed as being from the City of St. Louis. That IP is their standard outgoing IP for 42 various locations. As you can see the numbers are totally off the chart compared to typical traffic coming from many different ISP connections.
The city’s private security consultant did not want to characterize this as an attack. In fact, he said they can’t really track anything down because they have so many sites all using the same IP. I’ve been told attackers can sometimes “spoof” where they are coming from by giving a false IP address but apparently the type of tracking report my hosting company uses sees the real IP.
This leaves three scenarios.
Some have suggested the city’s server just randomly messed up and began hitting a site by mistake, my site. Can you imagine the odds of that? Another is that someone from outside the city’s network hacked into their system so they could launch the attack on my site and do it through the city’s system. That would be a scary thought that someone could do such a thing but I’ve been told it is not out of the realm of possibilities. The other, more realistic, conclusion is that someone did make a malicious attempt to knock out my site from within the system of the City of St. Louis. As I stated above, I’m told they have over 42 locations using the same IP address from the firewall.
I’ve reported the abuse to SBC (AT&T), the city’s internet provider. I’ve gotten a response only to say they are looking into the issue. I’m not hopeful they will be anymore forthcoming with information than the city’s security consultant was.
My site was slowed to the point of nearly being shut down. Sadly, the attack affected about 50 other sites on the same server including all the other blogs on the STL Syndicate and the Arch City Chronicle. The extra 5gb of bandwidth used by this attack does not come free.
Someone probably got a pretty good laugh over the whole deal but it shows a level of immaturity and fear that is unacceptable. If you don’t like my views write a well-reasoned opposing view but don’t resort to criminal activity just because you don’t like the message.
– Steve
Steve:
My background is in telecommunications and networking. My original thought on this was that your site was being downloaded for offline storage, so that someone could for whatever reason have their own copy of your blog.
With the data you’ve provided, that is still a possibility. If you look at entries 3, 5, and 10, those are very clearly search engine crawlers. Crawlers search a site, and create a list of pages with associated keywords, so that when someone searches for one of those keywords, the search engine can then display a link to one of those pages. Depending on the specific algorithm for a crawler, it may or may not revist pages of which it already has an index, and it may or may not download the entire page for indexing (after which they discard the page because it is for searching and not a repository).
What I find curious is entry #2 from the sphere.com domain. It is unclear from the URL the purpose of the computer (user, search crawler, etc.) and it had a heavy amount of use. It is also odd that your logging registered the root domain, and not a specific machine name, such as xxx-xxx-xxx-xxx.isp.tld or localmachine.network.tld, and that the domain is registered privately (only the registrar can tell who owns the domain). I went to the site (sphere.com), and it appears to be a search engine. So, thinking that it was another crawler, I put in your name to search. With that amount of downloaded data, if the purpose of the download was to index your pages for search use, you should have been all over the results, but you weren’t even listed. My last thought on sphere.com is that this is your webhost and that this was internal traffic, which I wouldn’t know, but would expect their webpage to advertise that they host. So, you might want to also look into sphere.com to see why there is such a drain of your bandwidth.
It looks like your biggest users are by far these crawler, this mystery user from sphere.com, and the questionable use from the City’s network. It also looks like you have fans in Russia–the . r u domain. You’ve gone international; way to go!
I still think it is likely that someone using the City’s network was downloading your site, unless your host was accurate that all the hits were to your main page, and not various pages. In which case, I would agree that it was something hinky.
Ok, I’ve technobabbled far too much now.
[REPLY Thanks for the feedback Travis. When you look at the big log of hits you see the same one repeated in rapid fire succession — neither indexing nor downloading. It was a continual barrage of requests for the main page, 20 times per second. – SLP]
Sphere.com is a blog search engine:
http://news.com.com/2061-10803_3-6067608.html
I (fatdays.com) am Steve’s webhost.
Based on the Apache access log, it’s possible that someone using a computer on the city’s network made an innocent (stupid) mistake when writing a .NET program that resulted in an infinite loop of requests for the main index page of urbanreviewstl.com.
If it was a malicious attack aimed at shutting down this site, it’s odd that they haven’t come back to finish the job.
Sphere is a legit search engine: http://sphere.com/profile?docset=3675642
Can you discern if the City-based slurp bothered to read your robots.txt?
Also, I’ve heard of trojans/worms out there that will try little DoS attacks like this on the top few bookmarks in IE. It could be an infected workstation from someone that just likes to read your blog. It’s scary that out governments security expert has no tools at his disposal to profile unusual traffic like this and determine a source.
Crashing a web site purposely is just plain wrong. I don’t know if it was malicious or accidental. I can only hope there is a logical explaniation and that it was accidental. Either view could be supported I guess….same as with the Kennedy assination.
But Steve, you are back up and running, time to move forward.
Maybe this is a sign of larger security issues and
other maladies going on at 1200 Market – IT Central. This sounds, like a job for Elliot “U PAID FOR IT” Davis and co. As a taxpayer, it worries me that vital information is volatile to hackers, and the city consultant can’t pinpoint the problem. Every inbound and outbound packet can be tracked. Who are they trying to fool.
The attack is a denial of service attack.
It would be interesting to see how fast the City’s internet connection is, because the transfer of 5 gigs of data, and 20 page views a second, would require a considerably fast connection. Furthermore, it would take a very fast connection to bring Steves website offline. The attack could be a DoS with someone faking the IP to match one of the public IP’s that the City uses. Either way, this was an DoS attack, with the source possibly being City Hall.
The interesting aspect is that City Hall had an internet loss the same time the attack was conducted, which leads me to believe that the attacker used most of the City’s connection to bring down Steve’s website.
How do you know the City had an outage at the same time as the attack? Do you work for the City?
I would suspect that the City’s uplink to be an OC1, but suppose it could be a T3, but that’s a lot of users for a 45Mbps connection. I’ve also heard recently from a city network user that there have been outages.
Travis: An “OC1” (which does not occur in nature BTW) is essentially the same as a DS3 (“T3”). As for users per connection size, I think we need to remember that the City doesn’t allow it’s users to just sit and surf all day, so traditional ISP bandwidth ratios won’t apply here. I suspect the City is more like any other business, where surfing is allowed but regulated. If so, a DS3 is overkill: a 45 mbps internet connection would be a whole lot more than they would need – probably about 40 mbps more than they need 😉
I have a friend who works at 1200 Market, and she said there were no outages that day (hrmmm… If I were a Conspiracy Theorist, I would look at Doug Duckworth’s posting and start crafting my next book – “The Great Internet Attack Conspiracy” 🙂
[REPLY – I was told by someone the next day that on the 15th 1015 Locust had experienced internet issues. Again, 42 different sites use the same outgoing IP. – SLP]
I consulted a bit with Steve after he first announced this. One thing that hit me as absurd is the city tech guy said:
“The city’s private security consultant did not want to characterize this as an attack. In fact, he said they can’t really track anything down because they have so many sites all using the same IP.”
That says two things: “I am an incompetent security consultant” or he is hiding something. They are incompetent if they do not keep records of IPs and destinations BEHIND the NAT’s IP (the single IP that is sourced for everyone in the city’s networks) or they are lying because they know what went on or refuse to look into it.
Either way, if that source IP from the city is the legit IP and the traffic came from the city’s network, it is still a DoS, whether intentional or through an accident.
We'r ed hardy outlet one of the most profession
of the coolest and latest ed hardy apparel, such as
ed hardy tee ,ed hardy bags,
ed hardy bathing suits, ed hardy shoes,
ed hardy board shorts ， don ed hardyt,ed hardy tank tops, ed hardy for women,
ed hardy swimwearand more,
ed hardy clothing. We offers a wide selection of fashion
cheap ed hardyproducts. Welcome to our shop or just enjoy browsing through our stunning collection available wholesale ed hardy in our shop.
our goal is to delight you with our distinctive collection of mindful ed hardy products while providing value and excellent service. Our goal is 100% customer satisfaction and we offer only 100% satisfacted service and ed hardy products. Please feel free to contact us at any time; we are committed to your 100% customer satisfaction. If you're looking for the best service and best selection, stay right where you are and continue shopping at here is your best online choice for the reasonable prices. So why not buy your ed hardy now, I am sure they we won’t let you down.