Tracking Search Engine Bots with Google Analytics on WordPress

Posted by Giuseppe Pastore - August 23, 2011 - Tools - 8 Comments

Being able to track bots and spiders on a website can give a deep knowledge of the way search engines interact with its pages, but, unfortunately, in most cases this information is not available since common javascript based stats softwares can’t monitor spiders (that don’t execute javascript) and often the webmaster doesn’t own server logs. Howevere, even when he owns them, without a log analyzer reading them isn’t comfortable at all.

Thinking about this issue, I wondered if it was possible to track bots using the King of stats software: Google Analytics. Surfing the web I found some french guys that have written a PHP script that exactly does want I was looking for (you can find it at http://www.web-analytics.fr/google-analytics-seo-comment-mesurer-les-vistes-des-robots-et-crawlers-sur-votre-site/; I don’t link since there are some spam uncensored comments in that post…)

I’ve given a chance to the script and tested that it works, but I also have tought that integrating it on a website wasn’t so immediate. So I sayd to my self: most of the websites are WordPress based, and WordPress uses PHP… Why don’t write a plugin to easy install the code without hacking template files?

From the idea to the result the step is short, and so we have now WP Bots Analytics: a very simple WordPress extension that could give you precious information on your website. Looking at its results, in fact, we can easily realize if:

  • There are pages visited very rarely (are they too far from the homepage? Do them have too few inbound links?)
  • There are other pages of great interest for search engines (Can I optimize them better? Can I use them to link to other?)
  • A few unwanted bots unnecessarily consume our bandhttp://www.stayonsearch.com/wp-admin/post-new.phpwidth (Should I block it via robots.txt file?)
  • If pages we wanted to block are still spidered (are directives in the robots.txt file or meta robots tag correct?)
  • Pages we want to be indexed are not visited at all (are we inadvertently blocking content?)

Knowing this kind of details can strongly help our SEO activity, but the greatest thing about that is we can monitor bots over time, look at trends and all the deeper set of data that Google Analytics offers.

To give you all these datas, Bots Analytics WP needs only 2 things:

  1. The Web Property ID associated with a new profile created in Google Analytics
  2. The first digits of the _utma cookie of the site on which you want to install it.

Let’s see how to get them.

Regarding the first one, in your Google Analytics account create a new profile and name it as you prefer (something like bots.site.com could be smart).

Take note of the Web Property ID.

For the first digit of _utma cookie, with Firefox you can get it from Tools>> Options>> Show Cookies. Find your site, expand and select _utma, take note of the first digits (till the first dot):

In addition to these two informations, you don’t need anything else.

Downloaded the plugin, you can install and activate it, enter the datas in the fields of administration panel (Settings>> WP Bots Analytics) and you have done.

In your Google Analytics account, you will see something similar to this:

Of course, now you can use all the capabilities offered by GA analysis to see which engine bots have passed on your pages or which pages a particular bot has (Googlebot, for example) visited.

Well, now it’s all!

You can download WP Bots Analytics from WordPress.org and if you need some help about it, you can add a comment on the official page on my blog.

I hope you’ll find WP Bots Analytics useful, and if you want to help me promoting it, links, tweets and shares will be very appreciated.

About the Author

Giuseppe Pastore

Graduated in Electronic Engineering, Giuseppe Pastore works as an SEO in an italian web agency. His personal blog about Search Engine Optimization is Posizionamento Zen. You can follow him on Twitter as @zen2seo or on Google+
11 comments
Custom Papers
Custom Papers

This is an absolutely great post. I recently decided to start seriously learning about positive psychology

adrian
adrian

very good post, i was really searching for such thing. I will install in on my wordpress. Thanks for sharing idea. My recent post Nie taki kredyt straszny

Ed@Zeek Rewards
Ed@Zeek Rewards

Thanks Giuseppe, I asked about this on webrank just today, because I'd been seeing the google bot visit in my stats counter progrogram, but I also wanted to know where it went afterwards and where it came from. My recent post How To Promote Zeek Rewards

depi
depi

This adds the bot visits to the website visits? I mean, that this manipulates the real website visits? Then this is not a good one plugin.. My recent post <h2>CommentLuv Premium is here!</h2>Offer expires on 4th October 2011<a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'> <img class='alignleft' src='http://cdn.commentluv.com/wp-content/uploads/2011/08/commentluv_premium_200.gif' alt='The CommentLuv Plugin' title='Check out The CommentLuv Plugin' /> Go to <a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'>CommentLuv.com to downloadThe version of CommentLuv on this site is no longer supported! it will stop working very soon!<h1> Hurry! <a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'>click here!</h1>

estate agent cardiff
estate agent cardiff

Huh. That's a pretty cool trick and very handy for checking what Google and other bots get upto while indexing your site! This is a great way to optimize ones site based on what and where bots go... Thanks for sharing :)

Intraday Tips
Intraday Tips

Excellent and informative tips.I like your post and it really gives an outstanding idea that is very helpful for all the people on web. Intraday Tips

commodity tips
commodity tips

very good post, i was really searching for this topic as i wanted this topic to understand completely and it is also very rare in internet that is why it was very difficult to understand thank you for sharing this, Regard Commodity Tips

franktown homes
franktown homes

It would be really beneficial to know which spiders are visiting your site as well as which pages these spiders are crawling. You could make adjustments to your strategy and probably add more links to your pages that spiders are not visiting that frequently.

Suzanne
Suzanne

Informative article on search engine using Wordpress and Google. Helpful tips for those who are starting out. Google search engine optimisation tools which are used for the purpose.

Claire
Claire

You made a great emphasis in tracking the search engine bots using Wordpress and of course Google Analytics. Its actually helping me in my wordpress website. Thanks. My recent post pc games