What is A/B Testing?
A/B testing (also known as split testing) is a controlled experiment that compares the performance of two different versions of a webpage by showing half your visitors one version of your site, and the other half the second version. By comparing the performance (measured by goals like signup rate or revenue) of each version of the webpage, we can empirically determine which version performs best and quantify the improvement rate.
A/B testing can be used to do a soft roll-out of a new design for a particular webpage, or to test different:
- Headlines and marketing copy
- Layouts and site workflows
- Designs and style elements
A simple example experiment could test a new homepage design or a different headline. 50% of visitors will see the original page, and 50% would see the new design. After running the test, you may find that the new design outperforms the original design’s revenue by 20% with 99% statistical significance. Or you may find that the new design hurts your conversion rate or any other goal you want to measure each variation on.
Creating Your First Experiment
To get started, create a new experiment and enter a URL for the webpage you want to run a test on. Once a URL is enter, the visual editor will load your original webpage into “Original” tab. To get started editing your webpage, create a new variation.
After adding a variation, the editor loads your original webpage in the new variation tab. You can switch to Interact Mode if you need to navigate inside the page to get to the element that needs to be changed. After you’ve located the page element that needs to be changed, switch back to Edit Mode to make changes to the page.
Click on the element to view the edit menu that allows you to make changes to that element:
Tracking Goals to Measure Performance
Goals let you define metrics to determine which variation performs the best wins in the experiment. Each variation's performance is tracked individually for each goal to make analysis possible from many contexts (i.e. a particular page change may decrease your engagement rate but increase signups).
There are four kinds of goals that can be tracked for experiments:
- Track URL Views - Track total or unique views of a URL pattern.
- Track Clicks on a Page Element - Track clicks on any element on your website.
- Track Engagement - Track how many unique visitors engage with each variation.
Adjusting the Experiment Frequency
The Experiment Frequency lets you choose what percent of your total visitors get enrolled in this experiment. For example, having a frequency of 25% means 1 out of 4 visitors to this page will be enrolled in this experiment. The other three will see the original page, and the fourth visitor will see one of the variations or the original.
Experiment Activation Settings
Experiments run on a user-defined URL pattern, such as:
- URL contains /plans
- URL matches regex \/profile-\d+
- URL does not match regex \/profile-\d+
- any URL (activates on every page)
In summary: if you're using Inspectlet without running any A/B tests (for session recordings, heatmaps, form analytics), use the regular loader. If you're running A/B tests (and also using other products), we recommend using the A/B loader to avoid any visible page flickering.
Deploying your Experiment Live
Once you've made the changes to your variation, click "Save". When you've added in all variation data and configured experiment settings like goals, experiment frequency and activation settings, click Deploy to push your experiment live. Once an experiment is deployed your website visitors will be able to see it immediately.
Viewing Experiment Results
The Results page lets you see how each variation in your experiment is performing. Results are available immediately after a goal has been completed: