"We got a 300% lift in revenues because we switched 'add to cart' button from green to orange!"
I read stuff like this all the time. It's not only horribly mis-informed but it can be damaging for business owners who execute baseless split-tests that lead to little or no real insight.
For ecommerce entrepreneurs who pride themselves on making the most out of every visitor, it's infuriating to be sold on - and subsequently invest in - conversion optimization tools, only to find after your first split test there were no real lifts at all.
The problem is, there's a good deal of information that is misleads the reader into thinking "If I just swap a headline, or change a button color, or 'fix' my call to action, the revenue will start pouring in!"
But that's far from true.