Ok, I’m going to let you in on a stupid mistake I made.

One simple little mistake knocked me off of the front page of Google and basically killed my site as far as the search engines were concerned.  In fact, it wasn’t really anything I did wrong, but one thing I didn’t do that caused these problems.

It started when I decided to install WordPress on the main site of my business, ServeSense.com.

Don’t get me wrong, that wasn’t the mistake, but that’s where the problem started.  Just so we’re clear, I’m a firm believer in WordPress.  It has changed my business.  In fact, soon I’m going to write a post about that in more detail.  But suffice it to say for now that I love WordPress and use it every day.

So what went wrong?

Well, I always used to install WordPress manually, by uploading all the files via FTP, creating a database and database user and password, then running the install script.  However, this time I decided to go the easy route.  Not that the manual route is hard, it just has a few more steps and I’m all about saving time where I can and wanted to try this other route to see if it really would make it easier.

The easy route I’m referring to is by using something called Fantastico.  And again, I’m not knocking Fantastico.  In fact, now I use it to install WordPress whenever I can.  It speeds up the process quite a bit.  But if I’m not mistaken, when I installed WordPress on ServeSense.com with Fantastico, I believe it was the first time I did it that way.  Fantastico is just a tool found in most cPanel installations that makes it easy to install quite a few different open source applications, including WordPress.  Even if you’re not very savvy when it comes to web stuff, you can probably install WordPress this way.

The Danger of Installing WordPress Through Fantastico

While Fantastico is nice and saves a lot of time, there is one significant difference when you install it this way.  In fact, it’s the only difference I’ve noticed.  As far as I can tell, everything else is the same.  But this one difference that I wasn’t aware of at the time is what caused these problems for me.

There is a setting in WordPress, under Settings -> Privacy that allows you to choose whether or not you want search engines like Google and Yahoo to be able to follow links on your site and index your site in the search engines.

When you install WordPress manually, you are given a choice of whether you want search engines like Google to be able to see your website.

Here’s what that screen looks like:

Ok, see that little checkbox that I have the red arrow pointing to?  Well, as of this writing, you don’t get that option in Fantastico.

While WordPress has that option pre-checked by default, assuming you do want your site to appear in search engines, Fantastico, for some reason, assumes you do not want your site to appear in search engines.  I have no idea why, since I’m sure most people do.  But by default, when you install WordPress through Fantastico, it’s as if they uncheck that box for you.

It’s not like it’s impossible to change or anything.  It’s a quick, simple fix (I’ll show you how in a few seconds).  But that’s assuming you know it did that.  In my case, I didn’t realize that.   I had always installed WordPress manually before that and the default setting was to have your site show up in search engines, which is what I wanted 100% of the time, so at that time I wasn’t even familiar with the options panel where you change that.  But believe me, I am now.

So if you’ve installed WordPress through Fantastico, here’s what you need to do.

Just go to Settings -> Privacy.

Here’s where to find that:

After clicking on Privacy, here’s the screen you’ll see:

See that red arrow?  It’s pointing to the default setting if you installed WordPress through Fantastico.  Obviously, all you have to do to change it is select the first option that says you’d like your blog to be visible to everyone, including search engines and then click Save Changes.

Very easy, assuming you know you need to do it.

But in my case, I didn’t realize Fantastico had set it up that way.

So months went by and I noticed my pages that were previously in the #1 spot on the first page of Google for certain keywords were no longer showing up there.  They kept dropping lower and lower until they were no longer on the first page at all.  And even the listings that appeared didn’t look right.  Instead of the page title and excerpts from the page like Google usually shows, all that showed up was my business name and the URL of my home page (instead of the page that was optimized for my keywords).

Obviously I was wondering what in the heck was going on.

I looked at my pages and I did notice one problem that I thought might have been the reason.  Before I switched my site over to WordPress all my links included www. (as in http://www.servesense.com), but after installing WordPress, the www. was missing.  As far as users are concerned, it’s not a big deal.  If someone types in www. in the address, WordPress just redirects them to the non-www. version.  But it’s better not to have those redirects going on if you can help it.

This isn’t normally much of an issue, but I had an existing site with existing pages indexed by Google and Google had indexed them all as having www. in the URL.  Now all those URLs were redirecting elsewhere (to just http://servesense.com – without the www. in the URL).  So that’s easy enough to fix.  You just go under Settings -> General and you can add the www. and then all your links include www.

So I did that and things just continued to get worse.

I don’t actually remember how much time passed (months, though) or the exact sequence of events, but at some point along the way I had a client come to me with this exact same problem.  He was using some of the web tools Google provides for webmaster and it was telling him there was a problem with his robots.txt file on his new site.  I took a look at his site and within a few minutes found the problem of the privacy setting and fixed it… problem solved (for him).

I talked with him a little and found out he had used Fantastico to install WordPress.  So now I knew to watch out for that.  But it didn’t click that I had used Fantastico on my own site.  Since then I’ve installed many WordPress installations for clients using Fantastico and one of my first steps after installation is to go into the privacy settings and make sure it’s set to be listed by the search engines.

However, for some reason, I didn’t think to check my own site for that setting.  In fact, I forgot that I even installed it through Fantastico.  So today I was thinking that my site needs some work.  I’m always getting my clients’ sites setup where both users and the search engines love them, but my own sites are often neglected.  I was just going to install a plugin for myself that I install for my clients that generates a site map and notifies the search engines so they can properly index the site.

Well, I have to give thanks to the developer of that plugin (Google XML Sitemaps).  I installed it and it gave me a message at the top of the page that said my privacy settings are currently blocking the search engines.  My jaw dropped.  Could I have had my site set to block the search engines for the past however many months, after checking that setting almost daily on clients’ sites?  Yes, yes I had.

I clicked Settings -> Privacy and there it was staring me in the face.  Obviously I quickly corrected this and sat there for a few seconds in disbelief.

In a way, it was a relief.  I was wondering why in the world my pages had been dropping out of Google ever since installing WordPress.  I knew if anything Google should have liked my site more.  So I was really puzzled as to why that was happening.  Now I had my answer.

Just in case you’re interested in what exactly this setting does on a technical level, it does a few things.  It adds a “nofollow” meta tag, it generates a virtual robots.txt file that returns a “Disallow: /” (which tells search engines not to index anything at all on the site), and also disables the pinging function that normally notifies pingomatic that you’ve updated something on your site.  For a little more detail, here is the WordPress page that explains the Privacy SubPanel.

So that’s what happened to me and why it happened.  Besides all the pages that were previously getting great ranking dropping out of Google’s search results entirely, the pages also show no Google PageRank anymore.  I’m curious how long it’s going to take for things to get restored.  The site has been around for a number of years, so I’m sure that will count for something.  But I also unknowingly had it blocking the search engines for months.  So we’ll see what happens.

I decided the first thing I’d do after fixing this setting and generating a sitemap is to write a new post so the search engines can get pinged and hopefully Google will start indexing my site again.  And what better topic to write about than how this happened?  Hopefully it will help someone else who is wondering why Google suddenly stopped liking their site after installing WordPress.  As I mentioned, it has nothing to do with installing WordPress, but just using Fantastico to install it.

Did you enjoy this article?  I’d love to hear your comments.