I’ve been having a helpful exchange with Matt Cutts about the difficulty I’ve been having with my site http://www.hystersisters.com being indexed in Google search engine. It all started because I posted a public letter to Google. I liken it to dragging rocks to the beach to spell out S.O.S. in the hopes of being rescued.I asked Matt if perhaps when I received a penalty in spring of 2006 and submitted a request for re-inclusion if perhaps I was released to Google purgatory instead of receiving complete grace.Matt’s reply: (Bless him!)

Kathy, I checked a while back and after leaving behind that particular link exchange, you got full grace. It would be helpful if you would do a follow-up post on your site to talk about what you’ve done with vbseo type stuff. My best guess is that in the process of trying to do whitehat SEO or streamlining urls, some change was made in the site architecture that block some pages in robot.txt or causes other issues. So a post that says “my urls used to look like bla. Then I made this change and now my urls look like foo.” might help.

My only frustration is not knowing exactly how to answer Matt’s questions but I’m going to give it a good try since I haven’t used vbseo and my urls haven’t changed (that I’m aware of - unless they changed from vb2 to vb3 by normal upgraded structure.)First: Robot.txt As you see I block such pages like member profiles, memberlist, attachments, register, search, picts, usercontrol panel pages. Over 80,000 pages are blocked for these reasons. We have over 127,000 member profiles blocked. Some threads are indexed and they are considered to be: vb2/showthread or vb2/post . Neither of these are blocked from the bots. I’ve checked the most recent threads in webmasters/tools: Test URLs against this robots.txt file. They are all allowed according to the tool.Site recent history and changes:I upgraded the Hystersisters website from vb 2 to vb 3 during the summer of 2005. Up until then and even after that point I was well indexed in Google, Yahoo!, AOL, MSN and the rest. In fact, Hystersisters was usually the #1 result for keyword hysterectomy and all phrases hysterectomy related.Sometime during the winter of 2005/2006 (I believe if I can recall properly) I added dp co-op links to the footer of my site and received a penalty from Google in the spring of 2006. Before anyone sits horrified at the notion I would do this, I innocently joined the link exchange, being told it was fine and dandy. When I learned of the penalty my first reaction was anger at Google.Then when I realized what the ramifications were for being connected to the bad websites represented in my category, I was royally miffed but not at Google. At myself for not checking it out more thoroughly to discover it is against “good webmaster practices”. At DP for not disclosing the possibly bad sites.I removed the bad link script. I submitted the reinclusion request to Google and received gracious replies from team Google. I thought all would be well.The following summer in 2006 I discovered that my pages in Google’s search results were still not showing up to reflect the number of pages on the site. Threads that were 2 months old were not showing up but threads 6 years old were there.Many of my comrades in the forum industry use vbSEO as a whitehat method to alter the thread urls in their forums. Many others do not. I chose not to use vbSEO because in the beginning it was an encrypted product. I wouldn’t allow it to be installed on my server because we couldn’t verify it wasn’t collecting data of some sort, sending it back to some “mother-ship”. I’m paranoid that way. Evidence: My server has a firewall. Added security measures. I comply with entities like the Hon Code folks. Credit card merchant compliances. Scan Alert. Hacker Safe.But lo and behold, vbseo Juan, came out with a vbseo sitemap for anyone using vbulletin. My web-developer installed it, configured it. And initially it seemed to help. My indexed pages grew to about 80K pages. Good deal. I was on a roll, heading towards indexing all 300,000 of our discussions. BTW, vbseo is no longer encrypted as of a few weeks ago. I’m still not convinced I need to install vbseo since most of forum owner friends that do not use vbseo have more pages indexed than those who do. Go figure. I think Juan has done a remarkable thing with the vbseo tool. I’m just not convinced its required. Fast forward another year to the summer of 2007. We changed the Hystersisters skin to make it less heavy in graphics. The skin we were using with the 2005 upgrade was beautiful but heavy. A new skin would make pages load quicker.Also in the summer of 2007 I signed up for scanalert (hacker safe) which required a firewall on the server. More server configurations to comply. With two servers for hystersisters.com, the connection between the two servers caused some DNS zone problems which the system admin fixed for us, trying to make sure we were good on DNS reporting.I thought we would doing well. Indexed pages were still between 30K and 80K. I had hopes of finding the magic egg to get all the pages indexed. I had hoped the lighter skin would help the bot get through the pages faster. If not the bot, it sure helped our visitors!And then October 2007:I woke up to discover another leak in numbers. We were hovering between 12K and 13K indexed pages in Google. Digging around I’ve found that the vbseo sitemap was no longer creating the sitemap to Google’s bot standards. I removed it from my webmasters/tools and have tried to find an alternative. Yes, vbseo upgraded their version of sitemap but it doesn’t include my older version of vbulletin. Yes, indeed. I am using an older version of vbulletin because of the extensive customizations done to the software for the benefit of my members.We do cool stuff with member’s hysterectomy dates, sending automated emails to them along their time line to provide additional information for them to discuss with their doctors, checklists of great info, reminders to “pamper the princess!” Yes, they sign up for “hysterectomy checkpoints” . Sorry, I digress. Its great stuff for our members but means that upgrading software is a challenge. The last upgrade in 2005 took 3 months of re-coding to add in all of our features.So, Matt, my urls never changed structure and neither did my site. I’ve closed the archive of my forums to send all visitors from the archive to the real thread discussion. I’ve tried to find a replacement for my sitemap. I’ve set the “crawl site at faster rate” at google.com/webmasters/tools suggestion. I’ve sent it back to normal. I’ve set it faster again.I’ve asked my system administrator to check the servers to make sure there is not a setting configured to block out the bot. According to the webmasters/tools the bot does visit. It just doesn’t stick around for long!I’m at a loss to know what to do or how to correct this dilemma. HysterSisters is considered an authority site within the niche and I would hope to find a way for our pages to be indexed to make sure women who need us, can find us.Thanks for listening, Google! And Matt, for your interest in my case, I call you my hero… especially if you can put me on the path to full indexing.

This entry was posted on Saturday, December 1st, 2007 at 9:58 pm and is filed under Random. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.