Champ car robots.txt

[nrc]

This one is for web geeks only. :)

I just went to archive.org to look at the old Champ car sites and I get a message that it's blocked because the site's robots.txt. For some reason the top of the Champ car site's robot.txt file looks like this:



# The '#' is a comment delimiter.
# To disallow all robots from accessing the site,
# un-comment the following two lines:

User-agent: *
Disallow: /

Why on earth would you want to shut every search engine out of your entire site? :confused:

[Spicoli]

Its those jacktards from **********.

They are beating us at the InnerWeb.

:gomer:

[coolhand]

i dont get it :cry:

[jonovision_man]

This one is for web geeks only. :)

I just went to archive.org to look at the old Champ car sites and I get a message that it's blocked because the site's robots.txt. For some reason the top of the Champ car site's robot.txt file looks like this:


Why on earth would you want to shut every search engine out of your entire site? :confused:

Agreed. If it's a marketing tool, you want to get it indexed by every search tool you can to drive traffic to it.

jono

[L1P1]

It appears to be working too. I just went to google and searched for "Kalkhoven" within champcarworldseries.com. Nothin'. Nada. Zippo.

Searching for "Champ Car World Series" will point you right to the home page thanks to the all the sites that link to it. But Google doesn't have a cached page.

[coolhand]

I liked CART's site for the 2003 season. 04 sucked. but this years is better

[nrc]

i dont get it :cry:
Exactly. Because of the way they have the site configured if you search for something on their site with Google, you don't get it.

[coolhand]

Exactly. Because of the way they have they're site configured if you search for something on their site with Google, you don't get it.

insane :shakehead

[Insomniac]

Is IIS configure out of the box with a defualt robots.txt file? If that's the case, maybe they never modified it and it is unintentional?

[racer2c]

Is IIS configure out of the box with a defualt robots.txt file? If that's the case, maybe they never modified it and it is unintentional?

Not that I can see. Although, they may not even be on IIS. It appears that the robot.txt is simply an internet standard for spidering exclusion. Someone would have had to manually create and exclude old Champ Car dot com pages. Why? Good question.

[L1P1]

Using netcraft.com, I can determine that cart.com has always been an IIS site. It doesn't seem to work for champcarworldseries.com. Perhaps Netcraft respects the robots.txt file.

I can get a single hit searching cart.com for "kalkhoven". So it looks like it wasn't always this way.

[Insomniac]

Not that I can see. Although, they may not even be on IIS. It appears that the robot.txt is simply an internet standard for spidering exclusion. Someone would have had to manually create and exclude old Champ Car dot com pages. Why? Good question.

They all redirect to the same place. The server is running Microsoft-IIS/6.0. If the server is using ASP pages, you can almost guarantee it is IIS. The robots file is a standard. My point was, on the default installation/setup of IIS, that file may be created automatically and they didn't know it should be modified or deleted rather than someone intentionally blocking all (good) spiders. I've never set up an IIS server, so I don't know it the file is there to begin with or not.

[racer2c]

They all redirect to the same place. The server is running Microsoft-IIS/6.0. If the server is using ASP pages, you can almost guarantee it is IIS. The robots file is a standard. My point was, on the default installation/setup of IIS, that file may be created automatically and they didn't know it should be modified or deleted rather than someone intentionally blocking all (good) spiders. I've never set up an IIS server, so I don't know it the file is there to begin with or not.

IIS doesn't create a robot.txt file.

[Steve99]

Could it be a copyright issue with who is/was running Champ Car's website? Or maybe they had a problem in the past with a malacious spider and went overboard?

[dando]

Why on earth would you want to shut every search engine out of your entire site? :confused:
Stealth marketing? :confused: :saywhat:

Either that, or the offshore company that created the site has no clue. :saywhat:

-Kevin