So I’ve jam-packed a great deal of work into the past few days in my quest to find the problem with my website’s Google indexing.
I don’t know if I have a complete solution yet but I feel very much closer to the end of the problem than before. So this is good news!
What I discovered on my end:
- My thread titles needed to be tidied up.
- My htaccess needs to be fixed to make sure my domain resolves at http://www.hystersisters.com./vb2/ and not http://www.hystersisters.com/vb2/index.php or other assorted similar urls. Indexing both constitutes duplicate content. Not something I did intentionally, in fact I thought I resolved the issue a year or more ago when system admin created a 301 rewrite in htaccess for the domain. But apparently it also needs to be applied to the forum directory, too.
- My meta descriptions were pulling the same description from the built in templates. Fixed to make sure they were unique to each thread.
- I double checked my robots.txt and found something more to add.
- I resubmitted my sitemap, removing more content from the config level.
- More memory has been alloted to php so as not to serve white pages when the google bot arrives on top of our already busy site.
My main question which sent me on this tizzified quest: Why isn’t Googlebot visiting my site? Was I penalized? Where have all the indexed pages gone? Long time passing. Where have all the indexed pages gone. Long time ago. (Apologies to Joan Baez)
We created a page to log Googlebot. To log the threads it indexes and to note the time it visits. I figured it would be a place to hang out and watch for any activity since I wasn’t seeing the googlebot on “who’s online”.
The page was created. A miracle occurred! The bot of Google was in the house!
We timestamped the log from Dec 1: Looking at my log page I see a list of threads and this message at the top:
Since Dec.1.2007 Google bot visited 25,417 threads!
Ah ha! Relief! The answer to my biggest mystery. I was asking the wrong question. The question isn’t “why hasn’t the bot been visiting?” rather, “why isn’t there evidence of the google’s bot work in the site: search function?”
Are my threads in Google’s index or not?
I do a simple site: search to find that 13,000 threads result.
But. BUT. BUT!!!!
If I do an advanced search, looking for keyword: “hystersisters com” in domain “hystersisters.com” I find 415,000 threads indexed!
It is possible I’ve been pulling out my hair for nothing much except a broken function on Google? Or not. It could be (based on my traffic stats) that some results are missing from SERP because this is where it all started. Searching for specific discussions I knew should be in the search index.
When you send my adsense gift for the year, will you send something wonderful? My favorite gift was the mood/light/radio from 2004. The USB kit wasn’t used much because my Mac OSX wasn’t happy compatible with some of it, but is so nifty I took it for “show and tell” at the family Christmas in 2005. I was deemed “cool” by the teen geeks in the crowd. Last year’s 2006 digital frame is still in the box because sadly, my Mac OSX couldn’t find a way to connect to it.
So this year I have a request. Please send a fuzzy hat. Put a Google logo on it. I’ll use it to cover my bald head from all the hair pulling I did this week making sure you hadn’t broken up with me.