How I Killed My Search Engine Ranking When I Installed WordPress

Ok, I’m going to let you in on a stupid mistake I made.

One simple little mistake knocked me off of the front page of Google and basically killed my site as far as the search engines were concerned.  In fact, it wasn’t really anything I did wrong, but one thing I didn’t do that caused these problems.

It started when I decided to install WordPress on the main site of my business, ServeSense.com.

Don’t get me wrong, that wasn’t the mistake, but that’s where the problem started.  Just so we’re clear, I’m a firm believer in WordPress.  It has changed my business.  In fact, soon I’m going to write a post about that in more detail.  But suffice it to say for now that I love WordPress and use it every day.

So what went wrong?

Well, I always used to install WordPress manually, by uploading all the files via FTP, creating a database and database user and password, then running the install script.  However, this time I decided to go the easy route.  Not that the manual route is hard, it just has a few more steps and I’m all about saving time where I can and wanted to try this other route to see if it really would make it easier.

The easy route I’m referring to is by using something called Fantastico.  And again, I’m not knocking Fantastico.  In fact, now I use it to install WordPress whenever I can.  It speeds up the process quite a bit.  But if I’m not mistaken, when I installed WordPress on ServeSense.com with Fantastico, I believe it was the first time I did it that way.  Fantastico is just a tool found in most cPanel installations that makes it easy to install quite a few different open source applications, including WordPress.  Even if you’re not very savvy when it comes to web stuff, you can probably install WordPress this way.

The Danger of Installing WordPress Through Fantastico

While Fantastico is nice and saves a lot of time, there is one significant difference when you install it this way.  In fact, it’s the only difference I’ve noticed.  As far as I can tell, everything else is the same.  But this one difference that I wasn’t aware of at the time is what caused these problems for me.

There is a setting in WordPress, under Settings -> Privacy that allows you to choose whether or not you want search engines like Google and Yahoo to be able to follow links on your site and index your site in the search engines.

When you install WordPress manually, you are given a choice of whether you want search engines like Google to be able to see your website.

Here’s what that screen looks like:

Ok, see that little checkbox that I have the red arrow pointing to?  Well, as of this writing, you don’t get that option in Fantastico.

While WordPress has that option pre-checked by default, assuming you do want your site to appear in search engines, Fantastico, for some reason, assumes you do not want your site to appear in search engines.  I have no idea why, since I’m sure most people do.  But by default, when you install WordPress through Fantastico, it’s as if they uncheck that box for you.

It’s not like it’s impossible to change or anything.  It’s a quick, simple fix (I’ll show you how in a few seconds).  But that’s assuming you know it did that.  In my case, I didn’t realize that.   I had always installed WordPress manually before that and the default setting was to have your site show up in search engines, which is what I wanted 100% of the time, so at that time I wasn’t even familiar with the options panel where you change that.  But believe me, I am now.

So if you’ve installed WordPress through Fantastico, here’s what you need to do.

Just go to Settings -> Privacy.

Here’s where to find that:

After clicking on Privacy, here’s the screen you’ll see:

See that red arrow?  It’s pointing to the default setting if you installed WordPress through Fantastico.  Obviously, all you have to do to change it is select the first option that says you’d like your blog to be visible to everyone, including search engines and then click Save Changes.

Very easy, assuming you know you need to do it.

But in my case, I didn’t realize Fantastico had set it up that way.

So months went by and I noticed my pages that were previously in the #1 spot on the first page of Google for certain keywords were no longer showing up there.  They kept dropping lower and lower until they were no longer on the first page at all.  And even the listings that appeared didn’t look right.  Instead of the page title and excerpts from the page like Google usually shows, all that showed up was my business name and the URL of my home page (instead of the page that was optimized for my keywords).

Obviously I was wondering what in the heck was going on.

I looked at my pages and I did notice one problem that I thought might have been the reason.  Before I switched my site over to WordPress all my links included www. (as in http://www.servesense.com), but after installing WordPress, the www. was missing.  As far as users are concerned, it’s not a big deal.  If someone types in www. in the address, WordPress just redirects them to the non-www. version.  But it’s better not to have those redirects going on if you can help it.

This isn’t normally much of an issue, but I had an existing site with existing pages indexed by Google and Google had indexed them all as having www. in the URL.  Now all those URLs were redirecting elsewhere (to just http://servesense.com – without the www. in the URL).  So that’s easy enough to fix.  You just go under Settings -> General and you can add the www. and then all your links include www.

So I did that and things just continued to get worse.

I don’t actually remember how much time passed (months, though) or the exact sequence of events, but at some point along the way I had a client come to me with this exact same problem.  He was using some of the web tools Google provides for webmaster and it was telling him there was a problem with his robots.txt file on his new site.  I took a look at his site and within a few minutes found the problem of the privacy setting and fixed it… problem solved (for him).

I talked with him a little and found out he had used Fantastico to install WordPress.  So now I knew to watch out for that.  But it didn’t click that I had used Fantastico on my own site.  Since then I’ve installed many WordPress installations for clients using Fantastico and one of my first steps after installation is to go into the privacy settings and make sure it’s set to be listed by the search engines.

However, for some reason, I didn’t think to check my own site for that setting.  In fact, I forgot that I even installed it through Fantastico.  So today I was thinking that my site needs some work.  I’m always getting my clients’ sites setup where both users and the search engines love them, but my own sites are often neglected.  I was just going to install a plugin for myself that I install for my clients that generates a site map and notifies the search engines so they can properly index the site.

Well, I have to give thanks to the developer of that plugin (Google XML Sitemaps).  I installed it and it gave me a message at the top of the page that said my privacy settings are currently blocking the search engines.  My jaw dropped.  Could I have had my site set to block the search engines for the past however many months, after checking that setting almost daily on clients’ sites?  Yes, yes I had.

I clicked Settings -> Privacy and there it was staring me in the face.  Obviously I quickly corrected this and sat there for a few seconds in disbelief.

In a way, it was a relief.  I was wondering why in the world my pages had been dropping out of Google ever since installing WordPress.  I knew if anything Google should have liked my site more.  So I was really puzzled as to why that was happening.  Now I had my answer.

Just in case you’re interested in what exactly this setting does on a technical level, it does a few things.  It adds a “nofollow” meta tag, it generates a virtual robots.txt file that returns a “Disallow: /” (which tells search engines not to index anything at all on the site), and also disables the pinging function that normally notifies pingomatic that you’ve updated something on your site.  For a little more detail, here is the WordPress page that explains the Privacy SubPanel.

So that’s what happened to me and why it happened.  Besides all the pages that were previously getting great ranking dropping out of Google’s search results entirely, the pages also show no Google PageRank anymore.  I’m curious how long it’s going to take for things to get restored.  The site has been around for a number of years, so I’m sure that will count for something.  But I also unknowingly had it blocking the search engines for months.  So we’ll see what happens.

I decided the first thing I’d do after fixing this setting and generating a sitemap is to write a new post so the search engines can get pinged and hopefully Google will start indexing my site again.  And what better topic to write about than how this happened?  Hopefully it will help someone else who is wondering why Google suddenly stopped liking their site after installing WordPress.  As I mentioned, it has nothing to do with installing WordPress, but just using Fantastico to install it.

Did you enjoy this article?  I’d love to hear your comments.

, , , , , , , , ,

19 Responses to How I Killed My Search Engine Ranking When I Installed WordPress

  1. Todd Heitner February 9, 2010 at 3:19 pm #

    Just a quick follow-up to this post. After fixing this problem, and writing this post, it took Google about 6 days to index this post.

    While I’m not showing any PageRank for any pages except the home page, that’s to be expected. Google doesn’t update that very often and it doesn’t necessarily reflect Google’s view of a page at that point in time.

    I’m back on the first page of Google for the search term I was referring to in this article. I’m currently in the #2 spot for those keywords and have been getting some traffic to that page.

    I acquired a new client this week through that page, which is why I decided to check back on things. It looks like things are basically back to how they were before I made that mistake.

    I just thought I’d share my experience in case anyone else finds themselves in this situation. It’s not the end of the world. But if your main or only source of traffic is search engines (which is not a good idea), then you’ll probably be in a pickle till it gets resolved.

    I would suggest doing as I did and, after making sure you’ve got the privacy setting corrected and that Pingomatic will be notified when you post something, to make a new blog post to get the search engines back sooner so they can crawl your site and see that they’re no longer being blocked.

    Hope this helps.

  2. Arun Basil Lal February 18, 2010 at 11:12 pm #

    I love Fantastico doing this for us. So the bots wont get the page before the blog is ready.

    I wonder why people don’t even look at every settings pages, there aren’t many.

  3. Todd Heitner March 12, 2010 at 1:52 pm #

    @Arun – Good points. I’m sure it does help with preventing the bots from coming and posting spam while you’re working on the site.

    I think it would be nice for Fantastico to give you the option of blocking search engines or not, though. Or at the very least let you know it’s doing that.

    I was used to installing WordPress manually and WordPress currently gives you an option when you install it as to whether you want to block search engines or not. I may be mistaken, but I think that’s fairly new. I believe it used to just allow search engines to crawl the site by default.

    Anyway, I’m not saying it’s a bad thing that Fantastico blocks search engines by default. My problem with it is that they are altering the default installation settings for WordPress without notifying you.

    Inexperienced users who are trying to get started blogging, hoping to get some traffic from search engines, are not going to know there is even a setting that controls this and probably aren’t going to be poking around in settings they don’t understand. A simple checkbox during the installation would be nice.

  4. Benson March 15, 2010 at 1:48 pm #

    I have noticed for quite some time that wordpress sites even on its own domain, don’t get added very well on search engines compared to conventional websites. It doesn’t matter if you select the privacy setting you mentioned in your article, search engines seem to bypass blogs and its much harder to find a website on keywords if they are set-up on a blog.

    My past websites with traditional html coding would get excellent search engine ranking and even older sites still exist on first and second page searches. This all ended when I started putting sites together with wordpress. With wordpress my site does not come up in good ranking no matter how many pages or keywords are added. It seems google is configured to avoid blog style pages, and mainly refer to these in blogsearch.google.com But how many people really use blogsearch.google?

  5. Todd Heitner March 17, 2010 at 4:50 pm #

    @Benson: That’s strange. I haven’t noticed any issues with WordPress sites not getting listed with search engines. I’ve had pretty good success with WordPress sites. In fact, to me it seems like they have better success than static sites.

    Everybody I’ve talked to says Google loves blogs, and my experience has been the same.

    Besides the fact that the content is well organized and tends to be updated frequently, by default WordPress pings pingomatic (that is, if you don’t install it through Fantastico), which notifies the search engines that you’ve updated something and usually gets them back to crawl your site for updates.

    I have Google Alerts set up to notify me when any new listings show up in Google that contain my business name, and my title tags all contain my business name, so I get notified as soon as any of my posts show up in Google. The last post I made, I got a Google Alert e-mail within 5 minutes. So that means Google got right over there and indexed the new page as soon as it got the ping from pingomatic.

    So to me, it sounds like there could be a problem of some sort with your configuration or something. It seems like if you have your privacy settings correct, and as long as these aren’t spam-type sites, that Google should be loving your new sites.

    Of course, I’m mainly referring to getting listing. Getting good rankings can take some time. It has a lot to do with the competition for the keywords you’re going after, as well as the quality of incoming links. I don’t think having a WordPress-based site is a magic bullet that’s going to automatically give good rankings or anything, as some people seem to think, but I don’t believe it would hurt either.

    I have noticed more and more spammy blogs popping up, so it’s possible Google might get a little more skeptical of new sites until they prove themselves. But I haven’t had any issues with sites not getting indexed or anything.

    If you have some WordPress-based sites that aren’t getting indexed, I’d be glad to take a look at them for you to see if I can spot anything that might be causing issues.

    Oh, one other possibility that could potentially be an issue is over-pinging. I don’t actually know if this is something to worry about or not, but I know there are plugins out there to keep your site from pinging too often.

    My understanding is that WordPress sends a ping every time you click the Update button on a page/post. If you’re just setting up a page or post on a new site and are trying to tweak things, that can sometimes mean hitting Update every couple minutes and then looking at the site to see how it looks. If it’s sending a ping every time, it can look like you’re sending too many pings.

    As I mentioned, I really don’t know what impact that can have. The maker of one of the plugins that prevents over-pinging claims it can get your site blocked, but I really don’t know firsthand. I usually don’t worry about it personally, but if I knew I was going to be making tons of updates, I might install the plugin.

    Just some thoughts. Again, I’d be glad to look at any sites you’re having trouble with to see if I can help you find the cause.

  6. Todd Heitner March 30, 2010 at 12:56 pm #

    Just a quick update. It appears Fantastico has been updated and now it no longer blocks search engines by default. Now its default behavior is the same as when you install WordPress manually, to allow search engines.

    Hopefully that will prevent this scenario from happening to other people. Of course, if Fantastico hasn’t been updated on your hosting yet, you may still get the old behavior. So it’s definitely still worth checking your Privacy setting after installing.

  7. Tom Coghill May 16, 2010 at 2:02 am #

    Hi Todd I owe you big time. I was changing a major site from html to wordpress and we used fantastico and http:/healthrecipes.com went from number one for the key words “health recipes” to not even in the ranking, lot of traffic missed and I was lost. Today you gave me the solution. Thanks for all the work you put into this post. I reset the privacy and changed the side bar on the workpress so google will reindex the site.
    Thanks
    Tom

  8. Dave July 27, 2010 at 11:58 am #

    This is very helpful i have been setting up wordpress site and http://www.easytojuice.com i am really having trouble doing that manual thing.. thanks to fantastico making things a lot more better!

  9. Jay August 5, 2010 at 1:50 pm #

    Very interesting post. However, I have used both ways of installing WordPress in the past and haven’t had a problem. Perhaps, the way fantastico deals with your installations is based on your Web host’s settings. Anyway..so sorry to hear about your experience. Makes me wonder how I would have coped if it happened to me.

  10. Todd Heitner August 6, 2010 at 5:50 pm #

    That’s possible. Or it could have to do with the version of Fantastico. I’m thinking they may have changed the setting at some point.

    I was glad to see that my site recovered after not too long. Fortunately I wasn’t too dependent on traffic from Google for my business at the time. But after having switched the site to WordPress and getting that privacy setting fixed and occasionally posting things, I’ve seen really favorable results from Google.

  11. SAP Business One September 15, 2010 at 4:06 pm #

    Todd, thanks for your post. This morning, in doing a keyword analysis for a client, I noticed that they had lost all rankings in the last 10 days since I migrated them to WordPress. Wow, what a knock. After several hours, I realized what had gone wrong after researched every other possible cause. I had block the SE’s intentionally because I used another domain to host the site while I built it and didn’t want it indexed but forgot to unblock it when going live with the new WP converted site. Yikes. So my strategy for repair is essentailly the same, I’ve created some bookmarks, rss feeds, new content and pinged them all. Within minutes I can see things being reindexed but still have no ranking for the primary keywords. Ouch.. damage control time, looking for solutions. How long did it take for you to get your rankings back? How much damage control did you do beyond the initial steps you mentioned?

  12. Todd Heitner September 21, 2010 at 12:35 pm #

    @SAP Business One:
    Oh man, that stinks. You’re right, it’s a different cause but the same problem.

    I don’t know exactly how long it took before my pages started showing up where they were ranked before, but I know that after about 2 or 3 weeks they were there. But I wasn’t checking it every day or anything either. I just know that when I checked back a few weeks later it was there.

    And as you probably know, there’s no way to accurately predict what Google is going to do. How quickly they restored my ranking may be no indication of what they’ll do with the site you’re working on. But at least it gives you something to go on.

    I really didn’t do any damage control at the time other than fixing the problem and then making this blog post to get Google to come back to the site. I might have made another blog post between this post and when I noticed my rankings back, but I don’t remember.

    One other thing to pay close attention to, especially before any more time passes, is to see how URLs were indexed in Google, specifically whether the www. was included or not.

    If the URLs in Google have www. (http://www.domain.com) but your URLs in WordPress resolve to just http://domain.com then my understanding is it’s viewed as a different page the same as if it were on a subdomain and won’t transfer the ranking the page had before. That’s another mistake I learned the hard way.

    Even though there were some bumps in the road, I’m still very glad I switched to WordPress.

  13. Joe March 9, 2011 at 9:31 am #

    Hi,

    Just had the same problem with 6 PR 5 sites I have bought over the last 3 weeks.

    The Fantastico issue (it is a problem I think as they don’t inform you) is still live. These sites were across two hosts and both had the blocking of the SE’s as default.

    I’d worked out the problem before finding your post but wondered if I’d done any long time damage.

    You’ve reassured me!

    Thanks

    Joe

  14. Jane Mountrose April 18, 2011 at 4:52 pm #

    Thanks for this information, Todd. I know you wrote this last year, but it is still relevant. I just noticed this problem on two of our sites and I have no idea how it happened, other than that we just updated wordpress on one of the sites. I hope google will come back soon.

    It maker me wonder how many people have this problem and don’t even know it.

    We’ll take your advice for adding a new post.

  15. Nell July 29, 2011 at 11:52 pm #

    I just bought a PR 4 domain and got that problem 🙁

    I wasn’t realized that i turned off the SEO and did repaired that by changing the privacy setting.

    Anyone knows when can I get back the PR 4 ? I bought the domain for high price and just bought it because the PR. It’s useless if the PR doesn’t come up

    Hopefully there’s some hero can help me ?

    Thanks for the share anyway Todd

  16. Richard Cummings January 27, 2012 at 12:09 pm #

    Todd, I was searching for exactly what that checkbox does technically and you provided that answer and a great story with it. Glad to hear you got your sites back in the SERPs. Cheers, Richard

  17. Jennie September 14, 2012 at 1:06 am #

    Read through your entire article, it was really informative. When I started a blog, I tried blogger and it just felt like so much reading or googling how to do things. So I switched over to WP, and liked the greater variety of layouts. So started working on my blog with on WP, then needed some more info on a particular narrator and voila, omg, there’s my blog on pg.1 of google search. I was so surprised. WP has a button that says ‘publish’ whereas, blogger just publishes after you click post. I had no idea it was going on the web. So I went back to blogger and started fixing that blog and adding to it.
    What I can add, to your article is, if you hit ping o matic to soon after you last did it, (I have a link on my tool bar) it will popup and tell you, you already pinged, & to try x mins. later. So if updating, your blog is pinging them,wouldn’t you get a message to wait. So maybe, google is using a different ping service. I have some doubts about the keywords, and google. They change their algorithm often and they do look for keywords, but you don’t need as many as in the good old days. You just need the right ones. I tried once in a post, to not mention my ‘keywords’ and came up on pg.2. went back later and made some changes, and it comes up on pg.1. I’m thinking of not changing to WP, but reviving the blog I started, same topic, different genre. I worry if I leave blogger, I’ll lose that pg.1 spot. I am not good like you at adding/changing things in the html, or anything about the www you talked about. So I’m grateful for your article, it’s warned me of pitfalls to watch out for. Thanks

  18. JimJohnJimmyJimmyJimJohn January 24, 2013 at 3:38 pm #

    Todd,

    Thank you so much for this blog. I installed a new WordPress site through Network Solutions portal and the site did not come up at all. Then I insatlled XML sitemaps plugin and it said my privacy settinsg were set to off. I tried changing it, but it wouldn’t save my settings. I read your blog, you said where the settings were. My version had it in two different areas. I didn’t know about it being under “Reading” Thank you, thank you, thank you.

    Yours,

    Jim John Jimmy Jimmy Jim John (Yes, that’s my real name. Hey, that’s what they gave me)

Trackbacks/Pingbacks

  1. Fantastico Installer and WordPress Privacy. Watch it | WordPress - February 20, 2010

    […] has come to my notice that, some people including, but not limited to Todd, forgets to change their privacy setting and hence, their blog seldom gets indexed, no matter how […]

Leave a Reply

website statistics