Experiments performed live on the web have major advantages over laboratory experiments. For one thing, the experimental "subjects" are simply actual visitors who are arriving on your site. The sample is the audience at the time the experiment is being run. The experiment can be performed transparently to the visitor. These factors provide excellent external validity. What this means is that your results will be highly applicable to your site since they were discovered on your actual site. It is also the case that you can run an experiment fairly easily using some modest CGI programming.
Page Experiments
Suppose you are trying to decide between page version A and page version B. Perhaps version A is the original and version B is a proposed new page design that is supposed to increase the number of links that visitors visit. There are two ways to create an experiment with these two pages. If this is the home page within a directory, or the home page of your entire site, you can replace the file index.html with a CGI script. You can view a sample page experiment script written in Perl.
Another way to go about this is to have a batch job which runs every 20 minutes, say. Lets say the page in question is called x.html. This job could work like this:
copy a.html to x.html
copy b.html to a.html
copy x.html to b.html
If you study these statements, you will see that when the batch job is rerun 20 minutes later, a.html and b.html are reversed back again. Every 24 hours you could reverse a.html and b.html one extra time so that each one is featured in a different 20 minute period than the day before.
To make sure you uniquely identify which visitor saw which page, create a single pixel transparent gif called a.gif and place it at the end of a.html. Likewise, do the same for b.html using a tiny gif called b.gif.
Banner Ad Studies
Here the simplest way to go is to use a CGI program in place of the banner referecnce:
<IMG SRC=/cgi-bin/banner-randomizer.cgi>
A sample version of banner-randomizer.cgi can be seen.