You can always choose to skip the science and go with your hunch, or to use just observational data. online controlled experiment, is the only scientific way to establish a causal link between our (intended) action(s) and any results we observe. This is where A/B testing comes into play, since an A/B test, a.k.a. However, knowing which actions lead to improvements is not a trivial task. In many online marketing / UX activities, we aim to take actions that improve the business bottom-line: acquire more visitors, convert more visitors, generate higher revenue per visitor, increase retention and reduce churn, increase repeat orders, etc. To explain it properly, we need to take a small step back to get the bigger picture, first.
Common mistakes in using statistical significance in A/B testing.Common misinterpretations of statistical significance.Significance in non-inferiority A/B tests.What does it mean if a result is statistically significant?.The first part where I explain the concept is theory-heavy by necessity, while the second part is more practically-oriented, covering how to choose a proper statistical significance level, how to avoid common misinterpretations, misuses, etc. This is not my first take on the topic, but it is my best attempt to lay it out in as plain English as possible: covering every angle, but without going into math and unnecessary detail. The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing.