How many impressions were you running if you were able to measure a 2% difference?
back
2 comments
If you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test
Check this tools[0] and this article [1]
[0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;8...
[1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html
When we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked).
Most of these numbers were measured after at least a week of leaving the experiments running.
We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.