[Seaside] [ANN] A/B Testing

Paul DeBruicker pdebruic at gmail.com
Wed Mar 31 22:19:01 UTC 2010


Hi - 


I've worked up a system based on the references at the end of this email
to make it easier to do A/B Testing of the efficacy of elements on a
page in Seaside.  Like Google Website Optimizer, ABingo, or Vanity, but
in Smalltalk for Seaside.  Right now it only works if you have a user
you can assign the variation (A or B). I think it could be made to work
with cookies, but haven't looked at it in depth, as users can log in
from many places and presumably prefer a consistent appearance. 

It tracks how many times each variation is shown and how many times a
separate action is taken after the variation is shown. The action can
be in any of the components in your site.  Over time it then calculates
whether there is a statistically significant difference between the
two (Using the G-Test), and tells you which one is better.  Once a
winner is determined, statistically, the loser is no longer shown to
anyone. It records which user is shown which variation in a class ivar.
It records all the tests in a class ivar.  
 

 a comprehensive example is:


renderContentOn: html
"This example alternates between two logos, and records when a user
clicks the button to indicate that they like something.  The logo shown
follows the user if the user is created prior to the #renderContentOn:
method. In this example the user is created in the render method, so you
get a new logo when the refresh is called. Creating the user in the
initialize method makes the logo follow the session. Creating the user
when they signup makes the variation follow them throughout their
experience until a winner is declared. "

	user:=ABUser new.
	html html: (ABTest showVariation: 'sample'
			   a:(WARenderCanvas builder render:[:h | h
			   image
			   url:'http://www.seaside.st/styles/logo-plain.png' ]   )
			   b: (WARenderCanvas builder render:[:h | h
			   image
			   url:'http://www.seaside.st/styles/esug.gif' ]   )
			   user: user). 
	html break.
	
	html form
	with:[
	html submitButton 
		callback:[ABTest recordConversion: 'sample' by: user. 
			html javascript refresh];
		with: 'I like this one'.
	]



Some questions I have are:
	1. Is this an appropriate place for WARenderCanvas builder
	render: or should I use something else?
	2. Should I be using Announcements for this somehow?
	3. Is there a need to put some kind of lock on the class ivars
	in the recordConversion:by: method or the
	showVariation:a:b:user: method?  I can imagine the ivars getting
	hit at the same time eventually, but don't know if Smalltalk
	just handles that sort of thing.  
	4.  Is there anything that obviously needs attention?

I wrote some class comments that may be helpful and provide some ideas
for other things to add, but I think it is functional at a basic
level.  There is also ABDashboard, a component that shows you how the
tests are doing.  There is an example of the dashboard output in that
class.  

The squeaksource repo is: 
http://www.squeaksource.com/ABTesting.html

It is MIT licensed.  



Thanks

Paul


http://en.wikipedia.org/wiki/A/B_Testing
http://elem.com/~btilly/effective-ab-testing/
http://www.slideshare.net/patio11/ab-testing-framework-design-3296257?from=ss_embed
http://www.bingocardcreator.com/abingo/
http://vanity.labnotes.org/ab_testing.html


More information about the seaside mailing list