Forget Statistical Tests: A/B Testing Is All About Simulations | by Samuele Mazzanti | Jul, 2024

How simulations outperform traditional stats in that they are easier to understand, more flexible, and economically meaningful

Controlled experiments such as A/B tests are used heavily by companies.

However, many people are repelled by A/B testing due to the presence of intimidating statistical jargon including terms such as “confidence”, “power”, “p-value”, “t-test”, “effect size”, and so on.

In this article, I will show you that you don’t need a Master in Statistics to understand A/B testing — quite the opposite. In fact, simulations can replace most of those statistical artifacts that were necessary 100 years ago.

Not only this: I will also show you that the feasibility of an experiment can be measured using something that, unlike “confidence” and “power”, is understandable by anyone in the company: dollars.

Your website has a checkout page. The ML team has come out with a new recommendation model. They claim that, by embedding their recommendations into the checkout page, we can increase revenues by an astounding 5%.