Curated by: Luigi Canali De Rossi

Monday, May 24, 2010

Website Monitoring And Testing: How To Monitor And Test Your Web Site Performance

Sponsored Links

Has your web server enough bandwidth to meet your readers requests? Would you know if your hosting provider is a bit sloppy and if it lets your site become inaccessible to the public, even if only for a few minutes? How do you find out if your web pages load fast enough now that Google places ranking value also on page load times? Monitoring and testing your website is a serious matter for any professional web publisher, big or small.

Photo credit: alexsl

Just like you keep under tabs your car oil and fuel levels to avoid frustrating problems, in the same way you need to keep under check your own web site and of all the specific performance, load and speed indicators which can indicate whether your site is performing and serving your content in the best and most efficient way or not.

In this guide, MasterNewMedia tech director Drazen Dobrovodski, explores most underrated aspects of website monitoring and testing.

Find out how to troubleshoot potential web site performance threats such as bandwidth loss, potential server downtimes, or sudden web traffic drops.

These below, the specific website monitoring and testing topics covered in this MasterNewMedia guide:

  • Web server availability
  • Web site performance testing
  • Web site speed monitoring
  • Google Webmaster Tools
  • JavaScript-based web site monitoring tools
  • Log file analyzers
  • Image hotlinking
  • How to use the robots.txt file
  • RSS feeds serving
  • Geographic web site testing
  • Real-time web site traffic monitoring

This guide can be read and understood by non-technical people, and it is in fact prepared for those who need to coordinate website supervision tasks as well as for those who will actually perform them.

If you want to find out how to best test and monitor your web site performance, speed and reliability, this MasterNewMedia guide has all of what you need to know.

How To Monitor and Test Your Web Site

by Drazen Dobrovodski

If you have a web site, no matter how big or small, you need to keep an eye on several important performance indicators in order to keep it functioning as expected. Some of these points are obvious.

Everyone's major concern is the number of visitors they get.

Whether your web site is just an amateur blog with grandma's cooking recipes or a major shopping site, everyone wants to know how many visitors they had.

Other points like difference between traffic monitors and log file analyzers are less taken care of.

However, the interaction between different web server contents, settings and performance is a point that is not so frequently discussed.

In this guide I will try to shed some light on all corners of web site monitoring, both for professional and rarely-visited web sites.


How To Monitor and Test Your Web Server Availability


Obviously, if your web site is not available to the public, that means zero visitors.

Today it is hard to find a web hosting provider that doesn't boast with 99% or better uptime. The reality is that very few web hosting providers actually have third-party confirmation of such claim.

Even if you have an amateur blog this is a point of concern, but if you have a professional web site - where downtime means money - then it is crucial to have a reliable way to monitor your web site.

Here are two lists with a number of free services that will keep a regular check on your web site:


How To Test Your Web Site Performance


The above guides will help you monitor your web site for "blackouts". They will tell you when your site becomes completely unavailable.

However, a web site owner needs to keep in mind that there is also a very big "gray area" when it comes to web site availability.

What if your site is available but painfully slow?

Just about everyone experienced the situation when you click on some URL from someone's tweet or similar and it is opening very slowly.

Then you think: "Oh, never mind, I'll just come back later" and you just go somewhere else. After that, you never really remember to go there again.

The question is: "How many other visitors never came back because the site was opening too slowly?" A slow site can lose a lot of visitors.

There are several reasons why your site can slow down. Too much traffic is one reason and it is actually a pretty sweet problem that all web site owners hope to get one day.

You should also be aware that a web site will not load equally fast from all geographic locations.

To verify this claim, you can perform a test using Traceroute.

If you go here:
and you hit that "Trace" button, you will get a list that explains a lot about how the Internet functions and why it is a "net" of all nets.

All those web servers from the list are the machines that stand between you and the web site where you clicked the button.

The further the geographic location of the web server is from the visitor, the more machines will the code of the pages have to pass on the way to your visitor.

That is why you need to know what is your target audience and then test how your web site looks from their location. You can do it using That site has a whole list of web servers in various cities and countries from which you can test your web site.

There are two separate functions that most of those sites offer.

  1. Ping: a function that allows you to test if a particular IP address (or web site) exists and can accept requests. Ping also returns the time it took for the web server to respond.
  2. Traceroute: a function that - just as the name suggests - traces a route that your request takes through the web servers to reach the IP address (or web site) that you requested. Traceroute also returns the time it took the web server(s) to respond.

These two types of services will help you to check the response speed of your web site from as many locations as you need.

Another type of service you may find useful are online proxy services. Proxies are web-based services that let you access any given web site from their own web server.

Thus if one proxy server is located in New York City, you can see how your site is seen from New York City and so on for any other proxy server in any other location.

This is especially important if you have Google Ads (or ads from a similar service) on your web site and you want to see what ads are displayed in what geographic area.

Along with online proxy services, you can also set up a proxy server in your web browser.

Proxy servers will help you evaluate your web site accessibility from different geographical locations.

Obviously, the best time to think about using a proxy server is before the site launch, when you are still developing your web site and choosing your web hosting service.

This brings us to the next point.


How To Speed Up Your Web Site


The best time to think about your web site speed and thus the site coding and hosting is when site is still in the planning stage.

If your site is already up and running and only then you realize that you have problems with speed, the solution will not necessarily be fast or easy.

However, the above-mentioned services will help you to determine whether the problem is in the hosting / server performance or in the actual site code.

The subject of maximizing the speed of a website is very important and very broad so you can find a complete guide on speeding up your web site here.


Google Webmaster Tools


Before we get into the details of using web site monitoring tools, let's first explain a few terms that you will frequently encounter when it comes to web site analysis:

  • Hits: The amount of files served by the web server. If you have a web page with one image in it, then every visitor to that page will register as two hits. One web page (HTML file) plus one image called from it.
  • Pageviews: A request made for any web page.
  • Visits: The number of web page requests from the same machine (IP and browser) with no more than 30 minutes between each page request. After 30 minutes it counts as a new visit.

These three are the most interesting numbers for all webmasters. Probably everyone already knows about Google Analytics.

Another Google service which is less famous, but a must for a serious webmaster is Google Webmaster Tools.

Google Webmaster Tools not only will tell you about the dead links on your website, but also all issues that may affect your search engine rankings, which include meta tags, keywords, page loading speed times and much more.


JavaScript-Based Web Site Monitoring Tools Vs. Log File Analyzers


One important distinction to be aware of is that both Google Analytics and Google Webmaster Tools are JavaScript-based tools.

You register there, grab a small snippet of code and place it into your web pages. That code will register only the visits to the web pages by JavaScript- and cookie-enabled browsers.

What this means is that all "non-page" files such as images, TXT, PDF etc will not be registered on Google Analytics. Visits from browsers with JavaScript and cookies turned off will not be registered as well.

To get a full picture of your web site traffic, you need a log file analyzer such as AWStats.

AWStats is a free software that most web hosting services already provide as a part of their hosting package. Usually AWStats is not activated by default, so you need to log into your hosting account and activate it yourself.

Once you compare the data from Google Analytics and your log file analyzer, you will see that they are completely different.

This discrepancy of data is something that many webmasters wondered about.

In simple terms, Google Analytics can only register visits to web pages by "normal users" (JavaScript- and cookies-enabled). Log file analyzers tell you the whole story. How many images your web site served, how many TXT, PDF, CSS, XML or any other files.

Finally, log file analyzers will tell you the total number of "hits" - all files requested from the web server.


How To Troubleshoot Web Site Monitoring Issues


The number of web server hits easily outnumbers the visits multiple times. This can give you some unpleasant surprises.

You could be looking at your visitor number and think: "Well, this is a modest number" and the very next day your hosting service could be sending you an email stating that your bandwidth is gone or that your web site is getting too many hits and you are asked to upgrade to a dedicated server.

Web server slowdowns can be a tangible symptom of how hits disproportionate to visits and in that case some urgent actions are needed.

There is no universal solution in such cases. Web site monitoring can prevent such situations and if the problem is already present, then analyzing the log files will tell you what the problem is.

Let's mention three frequent issues and solutions:

How To Prevent Image Hotlinking


Image hotlinking is when someone uses image(s) residing on your server / hosting account. Then your web server ends up serving those images both on your web site and on all other web sites that hotlinked it. Your bandwidth and money is spent on other web sites.

The solution here is to use the .htaccess file to protect your files from hotlinking.

A code like the following will do the trick (remember to change that ""):

RewriteEngine on

RewriteCond %{HTTP_REFERER} !^$

RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)? [NC]

RewriteRule \.(jpg|jpeg|png|gif|JPG|JPEG|PNG|GIF)$ - [NC,F,L]


How To Use The Robots.txt File


Your web site will not be visited just by human visitors but also by web spiders or bots as well.

The problem is that these web spiders and bots come not only from Google, Yahoo! and other major sites.

Over here you can see a partial (always growing) list of all bots, spiders and other "user agents" that visit web sites. As you can see, their number is in the thousands.

Spiders and bots will not target only your web pages but just about any file that you have. The problem is that these spiders can be very rude.

I have seen cases where server log files clearly show spiders that requested 10 different pages from a web server in one second. Given the fact that each page also has CSS and image files this can cause serious web server performance issues.

The Robots.txt file can help to restrain the invasion of web spiders on your web pages. You can cordon off sections of your web site (like images or CGI-BIN folders), set crawl delay time, etc.

Here you can find a set of useful resources from Google on how to create a Robots.txt file that will fit your needs.


Do Not Serve RSS From Your Own Web Server


RSS feeds are very popular today. All major publishing platforms like Movable Type and WordPress support them.

However, the bad news is that ever increasing is the number of automated RSS spiders that are visiting RSS feeds.

Many web sites collect news and posts from as many web sites as possible and as often as possible. It is not rare that some RSS spiders visit your site every hour. This can get especially painful if you have several RSS feeds (one RSS for each category of your content is rather common today).

Let's say you only have five categories and one RSS files for each + 1 full RSS containing articles from all categories.

That means that just one RSS spider will be making 6 x 24 = 144 hits a day on your server. Add to it your human visitors and other RSS spiders and the number grows.

Professional blogs can get 3000 or more RSS hits a day.

The solution is to use a service like FeedBurnerm, which is a free service from Google. You just have to register, enter the URL(s) of your RSS feed(s) and then your RSS feed(s) will be served from FeedBurner's server. Not your hosting account.

Depending on the number of RSS hits you are getting, using a service like FeedBurner can significantly improve your web server performance.


How To See Your Web Site As Visitors Do


The experience of your web site visitors may vary depending on their geographical location.

Big web sites care about location a lot because a web site may serve users different content in different languages depending on user location (detected through their IP addresses).

Copyright is also an issue. Some content (like videos on YouTube) is unavailable for visitors from certain parts of the world.

However, even webmasters of small web sites could be interested in location because your Google Ads also depend on which country your web site visitors come from.

This is where the data from Google Analytics comes in. If you are getting a lot of visitors from some country, then you should check what sort of ads those visitors get.

Do they get any ads at all? Are you completely missing a monetization opportunity there?

To check ads for specific geographic locations, you can use some of the online proxy services described earlier in the guide.

However, if you have a more advanced web site and you need to perform such checks a lot, then a service like GeoEdge can help.


Real-Time Web Site Traffic Monitoring


Once a web site grows to some amount of popularity and starts generating serious income, it becomes important to monitor its performance in real-time.

Google Analytics is not good in this respect, as it displays data for your web site with at least a few hours of delay.

If your web site had a slowdown and you were losing 50% of your visitors in a matter of hours, you will find out about it too late.

A simple fact is that every web site has certain peak hours when most of the traffic and income is generated.

You don't have the same number of visitors at 6AM as you do at 8PM. If this slowdown happens in critical hours (which is often the case because web spiders and visitors can be at their maximum at the same time), then you can lose over 50% of the daily income in few hours.

This is where real-time web site monitoring comes in. Your visits are monitored as they come in and if something happens you can react immediately.

Two services that you can use to monitor your web site in real-time are:

  • Woopra, which offers a range of pricing plans from free to professional
  • eWebCounter, whose free plan has a larger traffic range than Woopra.



In this short walk through the issues of web site monitoring and web site testing I tried to cover issues starting from those that concern every web site owner from amateur to professional and then slowly moved up to those issues that belong to web sites with higher levels of traffic.

As the web perpetually grows and changes, the list of issues that you need to monitor will probably keep expanding, but this little guide should still remain quite useful as a checklist of things to keep in mind when running any web site.

Originally prepared by Drazen Dobrovodski and Daniele Bazzano for MasterNewMedia, and first published on May 24th, 2010 as "Website Monitoring And Testing: How To Monitor And Test Your Web Site Performance".

Drazen Dobrovodski and Daniele Bazzano -
Readers' Comments    
blog comments powered by Disqus
posted by Daniele Bazzano on Monday, May 24 2010, updated on Tuesday, May 5 2015

Search this site for more with 








    Curated by

    New media explorer
    Communication designer


    POP Newsletter

    Robin Good's Newsletter for Professional Online Publishers  



    Real Time Web Analytics