Thinking about this issue, I wondered if it was possible to track bots using the King of stats software: Google Analytics. Surfing the web I found some french guys that have written a PHP script that exactly does want I was looking for (you can find it at http://www.web-analytics.fr/google-analytics-seo-comment-mesurer-les-vistes-des-robots-et-crawlers-sur-votre-site/; I don’t link since there are some spam uncensored comments in that post…)
I’ve given a chance to the script and tested that it works, but I also have tought that integrating it on a website wasn’t so immediate. So I sayd to my self: most of the websites are WordPress based, and WordPress uses PHP… Why don’t write a plugin to easy install the code without hacking template files?
From the idea to the result the step is short, and so we have now WP Bots Analytics: a very simple WordPress extension that could give you precious information on your website. Looking at its results, in fact, we can easily realize if:
- There are pages visited very rarely (are they too far from the homepage? Do them have too few inbound links?)
- There are other pages of great interest for search engines (Can I optimize them better? Can I use them to link to other?)
- A few unwanted bots unnecessarily consume our bandhttp://www.stayonsearch.com/wp-admin/post-new.phpwidth (Should I block it via robots.txt file?)
- If pages we wanted to block are still spidered (are directives in the robots.txt file or meta robots tag correct?)
- Pages we want to be indexed are not visited at all (are we inadvertently blocking content?)
Knowing this kind of details can strongly help our SEO activity, but the greatest thing about that is we can monitor bots over time, look at trends and all the deeper set of data that Google Analytics offers.
To give you all these datas, Bots Analytics WP needs only 2 things:
- The Web Property ID associated with a new profile created in Google Analytics
- The first digits of the _utma cookie of the site on which you want to install it.
Let’s see how to get them.
Regarding the first one, in your Google Analytics account create a new profile and name it as you prefer (something like bots.site.com could be smart).
Take note of the Web Property ID.
For the first digit of _utma cookie, with Firefox you can get it from Tools>> Options>> Show Cookies. Find your site, expand and select _utma, take note of the first digits (till the first dot):
In addition to these two informations, you don’t need anything else.
Downloaded the plugin, you can install and activate it, enter the datas in the fields of administration panel (Settings>> WP Bots Analytics) and you have done.
In your Google Analytics account, you will see something similar to this:
Of course, now you can use all the capabilities offered by GA analysis to see which engine bots have passed on your pages or which pages a particular bot has (Googlebot, for example) visited.
Well, now it’s all!
I hope you’ll find WP Bots Analytics useful, and if you want to help me promoting it, links, tweets and shares will be very appreciated.
This is an absolutely great post. I recently decided to start seriously learning about positive psychology
very good post, i was really searching for such thing. I will install in on my wordpress. Thanks for sharing idea. My recent post Nie taki kredyt straszny
Thanks Giuseppe, I asked about this on webrank just today, because I'd been seeing the google bot visit in my stats counter progrogram, but I also wanted to know where it went afterwards and where it came from. My recent post How To Promote Zeek Rewards
This adds the bot visits to the website visits? I mean, that this manipulates the real website visits? Then this is not a good one plugin.. My recent post <h2>CommentLuv Premium is here!</h2>Offer expires on 4th October 2011<a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'> <img class='alignleft' src='http://cdn.commentluv.com/wp-content/uploads/2011/08/commentluv_premium_200.gif' alt='The CommentLuv Plugin' title='Check out The CommentLuv Plugin' /> Go to <a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'>CommentLuv.com to downloadThe version of CommentLuv on this site is no longer supported! it will stop working very soon!<h1> Hurry! <a target='_blank' href='http://www.commentluv.com?ref=clp-oldapi962'>click here!</h1>
Huh. That's a pretty cool trick and very handy for checking what Google and other bots get upto while indexing your site! This is a great way to optimize ones site based on what and where bots go... Thanks for sharing :)
Excellent and informative tips.I like your post and it really gives an outstanding idea that is very helpful for all the people on web. Intraday Tips
very good post, i was really searching for this topic as i wanted this topic to understand completely and it is also very rare in internet that is why it was very difficult to understand thank you for sharing this, Regard Commodity Tips
It would be really beneficial to know which spiders are visiting your site as well as which pages these spiders are crawling. You could make adjustments to your strategy and probably add more links to your pages that spiders are not visiting that frequently.
Informative article on search engine using Wordpress and Google. Helpful tips for those who are starting out. Google search engine optimisation tools which are used for the purpose.
You made a great emphasis in tracking the search engine bots using Wordpress and of course Google Analytics. Its actually helping me in my wordpress website. Thanks. My recent post pc games