By Brian Livingston
When we started opening up the Windows Secrets Lounge to Google and other Web indexes a few months ago, we didn’t realize how hard it would be to get the search engine gods to find all our pages.
Finally, we hit the right solution. Google now includes more than 60,000 pages from the Lounge — over half of our total discussion threads — with the rest soon to become available to searchers around the globe.
As you’ll recall from my Jan. 7 Introduction column, the old Woody’s Lounge — founded by WS senior editor Woody Leonhard in 1995 — moved to WindowsSecrets.com in late 2009. One of our goals was to make available to the whole world, via search engines, the more than 125,000 threads Loungers had written since 2001.
For most of the Lounge’s history, the discussion board was hosted on a series of underpowered servers. Years ago, the volunteer admins decided to ban any crawling by search engines to prevent resource overload. In 2009, however, Windows Secrets moved the Lounge to a screaming server and invited search engines to suck down all 700,000 pages at will.
Just making your site visible, however, no longer guarantees that search engines will list all your pages. We had to make several changes to files with names like robots.txt and sitemap.xml to get Google to index more than a few hundred threads. But last month, the search giant got the message and started gulping down 10,000 pages at a whack. (See Figure 1.)