Why I hate split testing (hint: it’s a waste of time)

by Derek Halpern | Follow Him on Twitter Here

Split testing waste of time

Split testing is hot right now.

Change a button color, and BAM! Conversions through the roof. Tweak one word and watch revenue fly in.

And everyone is doing it.

There are new great tools that make it easy for non-tech savvy people to create and monitor split tests and that’s what they do. They create split tests.

Here’s the big problem:

Even though I’m known for being the “conversion guy,” I hate split testing.

…And I believe split testing is a big waste of time for many new business owners and entrepreneurs.

Why I hate Split Testing – Even Though Split Testing Software Makes it A Cinch

Many people start split testing because they feel as if they have to do it. And when you watch this video, you’ll see the truth.

Keeping that theme in mind – “feel as if you have to do it” – I have a quick question for you:

Have you ever done something because everyone else was doing it? You heard how great that thing was and felt as if you had to do it?

Leave a comment and share an example.

If you enjoyed this post, get updates. It's FREE



{ 108 comments… read them below or add one }

Marius Fermi

Finally! Someone else that holds the same values regarding split testing as I do.

I honestly think right now that the world of split testing has entered the realms of “affiliate marketing” in the sense that there are plenty of people out there touting their successes and making it look easy but the majority of us just won’t even come close to that level of success.

I’ve tried it and I ended up putting way too much focus into split testing, other projects ended up suffering.

Reply

Simon Green

Agreed Marcus. You have to take the split-testing claims with a grain of salt. There are way too many “gurus” out there.

Reply

Prerna

TOTALLY agree!! I have no intention of split testing anything just because I feel I have to do it!! :) Thanks Derek for highlighting this and making that teensy bit of “fear of missing out” vanish:)

Reply

Mark Davie

I don’t entirely agree with you guys. Although Derek’s example worked in that case, what if your traffic is already at 2000 and you are comparing the 15% conversion optimization to a the 1000 visitor increase? You would be better off with the 15% instead.

When you think about it, taking 100 visitors to 1000, means you were comparing a 10 times traffic increase to not even doubling the conversion rate! I know that 1000 visitors doesn’t sound difficult but if a site only has 100 (in a day a week a month whatever) maybe it is a small niche where targeted traffic is not as easy to get as you assume. Also if you are trying to get 10 times more traffic, it might only be achievable by using unrelated linkbait that results in your conversion rate actually dropping.

I agree that the small guys need to focus on their product, their blog and other traffic-generating marketing methods. They need to focus on whatever gives them ROI. But, if you focus on getting ROI out of your conversion rate, you will find that buying targeted traffic can be a lot more effective (and easier!) than trying to find it yourself. You can put your ad in places that don’t require you to use linkbait to get clicks and you will find that you get even higher conversion rates.

I still think that generating “quality” traffic is important, but there’s got to be a balance.

Reply

Paul Back

Hey Derek

I couldn’t agree more – for most businesses and blogs split testing is a waste of time.

There are many other things to invest your time and effeort into that have much, bigger results – in the long and short term.

In my opinion, for most blogs and businesses that time and effort should go into active promotion and building your email list as well as improving your auto responders.

I really like your no BS style, its actionable and there’s never any filler.

Paul Back

Reply

Jane

I have been a victim of this split testing fever when I started out. Yes the tools that make it easy tempt me a lot to do split testing. And yes it is a waste of time trying to up the conversion numbers a little – while I can be working on robust traffic strategies to up my traffic!

Thanks for the reminder.

Reply

Ruth Schwartz

The contrarian in me loves the contrarian in you, Derek. But it is still more complicated than you claim. AND in the end do you not recommend split testing?

Reply

Derek Halpern

Did you watch the video Ruth?

Reply

Dan

Derek said only the money points should be tested, rather than wasting time testing different aspects of our websites

Reply

Scott Costello

I love the idea of split testing. It really does make perfect sense to try and develop the best conversion copy/format/etc…

However I agree with you completely on the point of split testing should only be done when the time is right. What is the point of split testing if you don’t have a lot of traffic? I mean will you even get enough results are conclusive?

It reminds me of those commercial where they say 9 of 10 doctors agree that the blue pill is better then the red pill. So did they ask 1000 doctors to find those 9 people…then just say 9 out of 10?

You need a large sample size to really have confidence in any split test. So new businesses should concentrate on getting that traffic first.

Reply

Jeremy

You meant whether they found 900 out of 1000 doctors instead of literally 9 out of 10 right?

Reply

Benji Walklet

Yeah, I thought this video was going to mainly argue that new business owners lack the right size audience for statistically significant split test results. I think Derek makes a good point, but the size of your audience should be the first reason why you split test or not in my humble opinion.

Reply

Dave Linabury

Totally agree, Benji. I worked at a national ad agency and doing split testing for clients with less than say 10,000 visits a day is kind of pointless—unless you’re very patient. With 10k+ (and preferably 100k+), you can see results in the split testing immediately. With low traffic numbers, it can take weeks or months to see anything resembling “meaningful data”.

My 2¢…

Reply

Rich Page

From my many a/b tests run for 100s of clients, small and large, all you actually need is 1,000 visitors per week to the page you want to test to stand a good chance of getting of a significant test result within a few weeks. But really it all depends on number of conversions you get per week, not traffic – media sites often don’t have many ‘conversions’ like sales, so that’s why you probably experienced that.

Rich Page

Reply

Derek Halpern

Rich: 1,000 visits a week? That’s it? Here’s results from a recent survey I ran:

87% of people don’t even get 1,000 visits a month.

Reply

Derek Halpern

This video argued this point: “Just because you can, doesn’t mean you should.” Most people don’t know how to run proper split tests. And if they can’t afford an expert, they’re better off focusing on their unique ability.

Reply

Simon Green

I have a couple thoughts here:

1. Split-testing is of low importance unless you have a critical mass of on-site traffic. Moving on, the copy will be more important than the colors… Derek have you found that as well?

2. Many people who build sites spend WAY too much time agonizing over their layouts or headlines before they have any traffic whatsoever. I know I’ve done this myself. It’s safer to install 10 new templates and play around with them than it is to accept your site design and start trying to build an audience.

Reply

Dave Linabury

Actually Simon, colors can play an important role in the call-to-action. It’s been proven that (at least in the Western world) green buttons actually do get more clicks than any other color (so why do so many insist on blue and orange?).

Also, low contrast color combinations—which designers seem to insist on—perform poorly as calls to action (e.g., 30% grey text on a 40% grey background).

Finally, about 8.5% of males are colorblind. If you choose certain blue-green combinations, they might not see your message at all! You can test your designs for colorblindness issues at sites like vischeck.com. (I have no affiliation with them).

DISCLOSURE: I’ve been performing usability tests for 18 years and teach at Michigan State University.

Reply

Derek Halpern

When it comes to colors, the only thing I recommend is this:

http://socialtriggers.com/best-color-for-conversions/

And I’ll answer Simon’s questions.

1. Split-testing is of low importance unless you have a critical mass of on-site traffic. Moving on, the copy will be more important than the colors… Derek have you found that as well?

Split testing is important if you can afford to hire an expert. But if you’re not an expert, you should focus on what you do best until you can afford to hire an expert.

You can run some split tests here and there, but don’t let it derail you.

2. Many people who build sites spend WAY too much time agonizing over their layouts or headlines before they have any traffic whatsoever. I know I’ve done this myself. It’s safer to install 10 new templates and play around with them than it is to accept your site design and start trying to build an audience.

I’m with you there. you’re better off picking one and getting started.

Reply

Ricardo

Split test overuse is a waste of time. Not a judicious split test by itself. But I must say that the subject line and email copy made me run to watch the video. Nice play.

Reply

Beatrix Willius

100% agree. Split testing is a tactic and like every tactic it needs to be applied to a strategy. There are dozens of tactics when it comes to getting more traffic. Guest posting, go to Quora etc. After looking for about 15 minutes at the things that are discussed at Quora I decided that this isn’t for me.

Reply

Adam LoDolce - Sexy Confidence

Funny, I’m split testing like a wild man right now – and this is much needed advice. Good stuff Derek.

Reply

Simon Green

Adam I just took a look at your site, very well done video, sir.

Reply

Jason

Split testing is a waste of time IF your site or email marketing is modern.
I worked on over 200 sites in my career and let me know tell you changing colors for buttons, colors of text and the % of increase is always tiny… If you had a click rate of 10% before after then change maybe 10.2%
Mean while is cost you more money to pay me to do this than to leave it alone.

How ever I have seen great success when you change formats. And what I mean by that is the image was on the right and now its on the left. The buy now button is above the fold versus not etc…
But I agree with everything you said Derek. Love the site I am always viewing your videos …. you need your own tv show.
From Uniondale so I am from your neck of the woods.

Reply

Freddyjr

aaahhh!! .. you got me good Derek!

Called me out! … LOL .. I was one of those guys who is consistently split testing and all that -> when my blogs are not getting HUGE daily traffic flow YET!! ..

Well I was one of those guys, at least until after reading this and watching your video … hehe

But you are right on! … you just gave me some good tips Derek! .. thank you bro!

I shouldn’t be spending so much time on Split Testing when Im still working on the Daily Traffic Flow to my Blog. There is no point, I see.

Anyways, thank you for all your valuable tips and awesome videos! ..

Much success!

Cheers!

Reply

Carrie Jolie Dale

Great point Derek! I stepped away from Social Triggers for awhile…happy to be back.

Reply

Kristin Savory

Derek- Thanks for knocking down the bs- and giving us the straight stuff for building our lists. I’m an acupuncturist and busy mom of two- and new to the blogging for biz scene. It’s easy to get distracted in all the hoopla. Love your no-nonsense strategies.
Looking forward to another Thursday- Thanks!

Reply

JeremyC

Hi,

Split testing my adwords Ads made it possible for me to raise CTR from 3% to 12%. I have even had Some Ads with CTR’s of 20%.

I think this post i meant to polarize the Market to get attention and keep readers engaged.

Reply

Derek Halpern

You obviously didn’t watch the video.

Reply

Henry C

Derek, or should that be “The Conversion Guy”? (Does anyone apart from yourself call you that?)

Forgive me, but I’m having a hard time taking you seriously.

For a start, you manage to contradict pretty much every point you make in this video. Indeed, you contradict the very title of the video by recommending split testing when the time is right. In one breath you say that young businesses shouldn’t split test, in the next you say that new business owners ‘absolutely’ should split test.

You appear to have missed the point of split testing entirely; you discount split testing which is employed to achieve small, iterative improvements leading to long term results, on the basis that ‘big wins’ rarely happen. You say that time should be spent elsewhere like in building traffic perhaps by using tactics like controversial blog titles that voice an unpopular opinion, regardless of whether you can actually support that opinion with any tangible logic. In fact, split testing doesn’t have to be time consuming. Efficient split testing is a tried and tested technique which over time, will produce improvements when carried out correctly. Of course, this only works if you’re waiting to achieve statistically significant results before starting new tests, which I doubt you are.

I wanted to read more, but the rest of your blog seems to be more “content for content’s sake”, propped up by the big old circle-jerk you have going on with all the supportive comments placed by fellow marketing bloggers.

So as to save you time, I’ve responded to myself for you, in the manner that you have to the others whom have disagreed with your opinions in the comments:

Derek Halpern: You Obviously didn’t watch the video.

Reply

Derek Halpern

Henry, I didn’t contract msyelf. I said that new business owners shouldn’t split test anything other than money points in their business. Not the day to day microtests that so many people are accustomed to running and wasting their time on.

And I would say plenty of people call me the conversion guy. That’s why there are more than 20+ site review conversion videos on YouTube that have amassed at least 100,000 views at this point.

But you’re the expert. ;-)

Reply

Tim Felmingham

Try running the same ad twice at the same time and see how long it is before their CTRs become the same!

Reply

Wendy Merron

Derek, I have a quick question related to a recent launch where I advertised on Facebook. I split tested my ads on Facebook, which helped increase opt-ins dramatically, but it never occurred to me to split test landing pages.
In a situation like this, would you recommend I do both or just stick to the money page?

Reply

Claire Bullerwell

Hi Wendy,

I’m just about to start ads on Facebook and I’m going to definitely split test my FB ads, but as long as my landing page converts over 40% I will be happy. Is this the right percentage figure to aim for (40%) or is this too high – anyone? Thanks,

Reply

Derek Halpern

If you’re spending a considerable amount of money on traffic, you should be testing your pages.

Reply

Tim Felmingham

I completely agree! I’ve created duplicate Ads several times and run them side by side – equal rotation with the same keywords, the same landing pages, the same bids, the same campaigns (AdWords). Often I will see very significant differences in CTR and CR (say 1.2% versus 2.2%) for a surprisingly long time — a couple of weeks or more, and that’s with clicks of a few hundred a day. The Ads are identical! It’s all about the sample size and it’s probably bigger than you think.

Reply

Remco Boom

I’d like to add that there’s no significant meaning in running split tests if you only have a limited amount of visitors.

With only a few visitors the results of a split test mean very little and could even provide you with a false result; pushing conversion even further down.

Reply

Brian Dempsey

Great point, Remco.

Reply

Brian Dempsey

Great video! Just this week I was asked during a kick-off meeting for a new website (a golf course / country club) whether we did “A/B Testing.” Of course, there were no details like, “Do you do split testing for AdWords landing pages to increase conversions of new members?” This was a question that was printed off the internet to ask potential vendors. It’s become a popular buzzword (like “content marketing”), but in many (most?) cases, like you mentioned, it doesn’t actually produce significant or long-term results.

Reply

Janice

Thanks so much, Derek! I have been telling my small business clients this for years (after learning it the hard way myself, lol). As the sign over my desk reads, “Better done than perfect.”

Reply

Robert Gregg

Very concise reasoning. Thank you. That makes a lot of sense.

Reply

Keith James

Hi Derek, great post but I don’t think I can agree with all of your points.
I always want the highest conversion no matter how much traffic I have. What is easier, making small tweaks to your offer or increasing your traffic ten fold?
Now lets consider organic verses paid. The biggest mistake businesses make with PPC campaigns is that they create a couple of ads and never split test. What a waste of money.
Shouldn’t we be using split testing to also increase our traffic? Are my headlines compelling? Do my visitors want a blog post or a video?
Split testing is not only used to increase conversion but to also increase traffic. This is the only way we can really know what our audience wants. Test everything and assume nothing.

Reply

Derek Halpern

Here’s the MAIN point you’re missing. Those small tweaks almost NEVER lead to a ten-fold increase. You’ll have to run 500 tests before you get results like that. And the time it takes to run 500 tests – especially when you’er just starting out – is better spent on anything else.

Reply

Star Khechara

YES!!! finally someone gets why I don’t constantly split-test! Full confession – I’ve never actually split-test anything and still built a 6 figure biz from the ground up in a very short time. I focus my efforts more on what my customers are about and what they love about me. I’m tech savvy but my approach is a soft-sell one – it works for me :-)

Hey, Derek more of the sexy stuff please, you’re so my ‘geek crush’ ;-)

Reply

Tatiana

Like always with you, this makes a lot of sense. Split testing is time consuming, so it wise to do it only where and when it is crucial.

Reply

Mridu Parikh

Great video Derek…makes a lot of sense! Thanks for sharing your insights and experience.

Reply

Meraid Griffin

I took some time out to watch and learn. What did I learn? Well, I had no idea what ‘split-testing’ was or indeed whether I was wasting my time or not. Thankfully, I seem to be working on building traffic with content and I’ve missed out on making this BIG error.

Thank you Derek.

Reply

Kyle Alm

Business cards? I feel somewhat naked without them, but I can say that they have ever really brought me a lead directly. But there is a trust factor that starts with the exchange that indicates you are invested in it.

Reply

Allison Rapp

Kyle, try leaving your business cards at home. That way, you can tell people are interested in you, “Gosh, I don’t have a card on me! Why don’t you give me YOUR card, and I’ll call you… is Tuesday afternoon good for you??

Puts the follow up in the hands of the person who is most likely to do it. Unless you’re the kind of person who doesn’t follow up. ;)

Reply

Kyle Alm

Allison, I have been without them for a couple of weeks now. I can’t decide on a couple of design changes, but I digress. I think that it’s an expectation, and I would kind of like to do without them, but I think they really do make everyone feel better. And there are people I would have forgotten if I didn’t have their card, and I can’t be forgotten!

Reply

Allison Rapp

Kyle,
Sounds like the perfect time to test the idea! I’m sure you’re still meeting people while you’re deciding on the design of your next card. Why not find out what happens when you take the initiative to follow up?

I am not sure what you mean by making everyone feel better. I haven’t used them for years and I feel just fine ;) I actually do have a few that I sometimes have with me and I only give them to people who really press me for them–and I have NEVER heard from a single one of those people.

I wouldn’t say don’t have them at all–I’m saying I do much better by taking the responsibility on myself to keep the connection and expand it.

Reply

Andrew HaHa

Hey Derek,
Excellent video. I agree 100%. I’m killing it with Facebook ads. More relevant traffic = more conversions. I’ve tried split testing before as well, but nothing brings me the results that I get just by sending more traffic.

And a shout out to Henry C, I also call Derek the conversation guy ☺

Reply

Liudas

Really great points about split testing :) For many newbies it’s really better to go after more traffic than just try to increase conversions.

Reply

Shana

Thank god for you Derek! I’m in a group watching everyone split testing everything. I thought I was the only one that thinks action is more effective than testing. I’m launching my website in a few days and split testing is the last thing on my mind.

Thank you for your smart insights and speaking the truth!

Reply

Daniel

Regarding the gurus who tell us about their 300% jump in conversions (I think about the weekly emails I get from LeadPages) or other things like that….

1. It’s reporting bias. We don’t hear about the 30,000 landing pages that week that ran split tests where there were no substantial differences.

Neil Patel wrote an article about this on the KISSmetrics blog a while back

2. It’s kind of stupid to run a split test if you only have 200 or 500 followers. As Derek mentions…. focus on getting your following up first, as that is a lot more productive.

3. Corollary: A lot of the gurus will tell you to split test your ad, email, offer, landing page, etc. on a small segment of your audience (perhaps 10%). Then take the winner and unleash that upon the rest of your audience. Hard to do that when you only have 500 followers.

Great video, Derek, very helpful. I’ve stopped focusing tactics videos as I’m just starting my business. This video is a subtle reminder that I’m on the right track.

Reply

Mary Planding

Well said Derek! Thank you! We get lots of requests / inquiries from clients almost hounding us to do split testing because they think it will make them tons more sales. The math says it all. I enjoy your naming them “Money Points” vs Google’s ZMOT. Much simpler concept for clients to grasp.

One client insisted on this, despite our showing him the math. He spent about $5K in testing with us with only a slight bump in sales, and didn’t cover the cost of the testing. When we finally got him to focus on traffic and his primary landing page, his sales more than paid for the work done. Then and only then did he finally understand what we meant.

Keep those great tips coming! You’re helping a lot of people.

And I’m really enjoying your podcasts too.

Reply

Chief Conversionista

Beer sucks is if the brewery stinks.
TV stinks if The actors are rotten.
A/B testing is rotten if The goals & methods are pointless .

So you can’t prove that split testing is flawed by showens bad ways of doing it.

Split testing is a wonderful tool for capturing visitor insights that were once hidden from you.

Reply

Tommy Walker

“Tv stinks if the actors are rotten”

This one made me lol.

Yeah, Derek, I agree, you should absolutely be testing your money pages that you’re driving traffic to (like you did with the website review videos) but also, you should be running tests for longer periods of time on the low traffic pages too.

http://conversionxl.com/how-to-do-conversion-optimization-with-very-little-traffic/

These kinds of tests take longer to run & because of that are very little maintenance. Set the test up & let it run in the background while you build traffic.

Once you’ve reached a statistically significant result, then call take what you’ve learned from the test & move on.

But don’t forego the testing all together, because what the hell have you learned about what gets your audience to convert as it’s growing?

How are you using your feedback loops to feed those test hypothesis after you’ve reached a significant result?

Yes, testing the Green vs Red button is stupid.

But it’s stupid, because it’s not based in anything real.

I wish you hadn’t made this video about why A/B testing is stupid for low traffic business & made it about an either/or scenario.

It’s not.

You had a great opportunity to educate on how to run a proper test with low traffic, but instead suggest people focus on traffic building, without even giving advice or links to advice on building traffic.

Reply

Derek Halpern

I’m not saying you shouldnt. I’m saying, just because you can, doesn’t mean you should. And when people have a traffic problem, their time is better spent working on a better foundation before worrying about split testing.

Reply

Tommy Walker

And I fundamentally disagree.

The time it takes to set up a value proposition test above your email form is 5 minutes.

If that value prop test runs for 3 months while you build traffic, by the end of your traffic building campaign, you’ll know which is a more significant result.

If you’re spending 3 months A/B testing button colors, yes, you’ve got better things to spend your time on.

But to suggest that knowing what value proposition resonates more with the traffic you’ve been building after 3 months isn’t foundational…

It’s not playing with a full deck. And it’s perpetuating ignorance that surrounds conversion testing as a whole.

Reply

Rich Page

Ah, a breath of fresh air Tommy :)

Derek Halpern

You’re right, you can set up that test and run it for 6 months. Or you can not waste any time “figuring out” how to set up a value proposition test – some people will be like, “what’s a value proposition?” – and just focus on what’s you’re good at , and what’s more important. Like a business model, base levels of traffic, a product, and etc.

Tommy Walker

And what of the people who’ve built something that is perfectly fine, but haven’t yet discovered the language that best describes what they do in a way that resonates to the little traffic they have?

Even if it’s under 1,000 visitors a month, making sure what they do is clear is incredibly valuable.

You could be the best at something, but if you don’t know what language lands – because you’re not testing – you’re going to waste a ton of time trying to build traffic that doesn’t ever get traction, because you haven’t figured out how to say what you do in a way people understand.

The solution isn’t to focus on other stuff. The solution is to understand the tools available to you.

But furthermore sometimes it’s not even a conversion problem, or a traffic problem… Sometimes (and more often) it’s that you’re trying to build traffic to a crap-tastic idea.

Testing, not on a website, but just by talking to people who might be potential customers, is going to save you a boatload of wasted effort (either split testing or traffic building) either way.

In any event, testing is more foundational than you’re giving it credit for & the video is in a roundabout way suggesting people build traffic to ideas that might not be viable.

Perhaps something on customer development would be a good followup?

Derek Halpern

Tommy: Learning how to set up a proper split test and waiting 6 months for a result is what I’d call a waste of time.

Spending that time to learn how to setup one split test means this: by the time you get results you’ll forget how to do it.

You’re better off driving traffic. Learning how to split test and then running one split test over a 6 month period is a sure-fire way to become a failure.

Jen

Thank you, Tommy!

Joshua Rivers

First, that “sexy” spot was a little … uh, awkward.

Second, I like this perspective. I’ve never done split-testing, but have thought about jumping in. After this, I will postpone and focus on building my audience and list.

Reply

Derek Halpern

I know, you’re telling me. I had to watch myself. After every video, I immediately take off my shirt. My video guy managed to make it look as embarrassing as possible. Ha.

Reply

Joshua Rivers

I guess you’re kind of known as the “embarrassing photo/video guy” as well as the Conversion guy. :)

Reply

Rok Sprogar

Split testing is like buying $200 running shoes. If you can’t run a single mile, they are a huge waste of money. But if you are a professional marathoner, you will burn through three pairs per a season.

Reply

Derek Halpern

Exactly

Reply

Mercedes Brennan

Thank you so much for this post, Derek! I just keep following your advice and that’s putting out valuable content. I have felt the pressure to split test, but I have not done so because… well… I want to keep writing good content and building my email list.

You have no idea how you put my mind at ease today. Thank you again for being such an original thinker and reminding us not to act like sheep. :)

Reply

Cristina

Derek,
You bring really valid points but I find the title of your post misleading.

Prioritizing split testing over more urgent to-do’s – yes, detrimental.
Doing split testing without valid hypotheses that come out of previous data analysis – yes, it’s looking for a needle in the hay.
Having little data to start with and hoping that a change of a button or a title will bring results – LOL, totally.

But done in the right context, considering the cost vs. benefit and based on previous data analysis it can be very beneficial. And fun. And help understand your audience.

Reply

Peep Laja

The content of this post is wrong at several levels: a) the language used, and b) understanding of what a/b testing is and how it fits in to the big picture.

First of all, hating split testing (your quote at 00:20) is like hating clients, hating revenue generation or hating marketing. Sure, you can hate all you want, but if your goal is to grow a business, it’s stupid.

I’m guessing your audience has a lot of people who’re just getting started, people who look up to you, so stating something like “split testing is a waste of time” is a dick move.

I get your argument that running split tests without much traffic is difficult as it takes forever to get to a statistically significant results (although you never mention that, but I assume you know). The flaw of your argument is that people are have to choose between split testing or traffic generation / other marketing. These are not contradictory at all.

In fact, if you have a low traffic website, and you actually know what you’re doing when running split tests, you know that a test should not be stopped before you have a decent sample size (AT LEAST 100 conversions per variation, better yet 250 or 350), you have good enough confidence level (95% or up) and there should be no overlap in error margins.

Now if you have a low traffic site, it will take perhaps 1,3 or 6 months to get to a point where your test results are actually valid. So – you set up the test once, and that’s it. Now you have all the time in the world to do other stuff.

Split testing is not something you do every hour of the day, every day – especially on a simple website like a blog. You set it up, and testing works by itself. Magic! So you have 99% of your time left for other marketing activities.

You ignored testing for 1.5 years? So all that traffic all that time could have converted significantly better, but you just wasted a bunch of traffic. Not very smart.

I don’t know what your average opt-in conversion rate is, but let’s say that it is 20% (the number you used in the video). And let’s assume that your traffic during those 1.5 years was a low 10,000 unique visits per month (prob way more, right?).

So 18 months x 10,000 = 180,000 visits * 20% conversion = 36 000 subscribers.

What if you had improved the conversion rate to 25%? 45 000 subscribers.

35%? 63 000 subscribers. (Almost double).

What if you had on average 20,o00 visitors / month? 40k? Numbers would be way higher.

Sure, it might have taken time to get to those conversion rate levels, but all in all you lost a) thousands of subscribers, b) a shit ton of learning about what works for your audience (actual data and insights, not survey responses about hypothetical action).

Split testing is about learning, validating hypotheses. Growing an audience is way easier if you learn about them along the way. If your test gets an uplift – that’s nice. But way more important is to get insights.

I started my blog (ConversionXL) just 2.5 years ago. I wrote all the content myself, did all the marketing myself – typical one man show. I got to 100,000 monthly readers in 12 months WHILE running split tests to gain insights about my audience. It’s not this vs that at all.

Another thing that was way off was making it seem like split testing and conversion optimization are the same thing. You can optimize your site for conversions even without split tests. In fact, split testing is the tip of the iceberg.

I agree with you that low traffic sites should not rely on split tests for learning. There are things they can do that get insight way faster.

All the qualitative stuff for instance. Super useful. I won’t get into it in detail here, but I would encourage everyone to read this to learn how you actually boost conversions when you have very little traffic: http://conversionxl.com/how-to-do-conversion-optimization-with-very-little-traffic/

Also I really did not appreciate that you make it seem that A/B testing is about testing random ideas. If you brainstorm a bunch ideas to test, you will inevitably end of up tests that provide little to no uplift. Mentioning stuff like button colors and what not is what gives people a wrong idea.

“Green vs orange” is not the essence of A/B testing. It’s about understanding the target audience. It’s about identifying friction, solving problems, addressing pains.

At 1:30 you suggest that you should run 20,30 or 100 split tests before you see big gains. It’s not about the number of tests really, it’s about whether your test is based on a data-backed hypothesis at all.

If you run a test that’s not based on data that gave you insight that lead to a hypothesis, you’re doing it wrong.

Split testing is about bringing actual data and science into marketing. Running your business based on opinions and creating split tests while dreaming in your office is not how conversion optimization is done. Running your business based on numbers is how marketing today is so different from the old days.

I get it that you intentionally want to create controversy and buzz, but please don’t do it at the expense of getting lots people to believe that split testing is a waste of time, evil etc.

Oh it’s happening, you can already see it in the comments of this article. “I have been a victim of this split testing fever”. *facepalm*

Reply

Derek Halpern

Hey Peep, what’s up?

First, I want to make clear that I think you’re an upfront dude. And you’re one of the better writers when it comes to split testing and conversion optimization. You also probably know a lot more about it than I do.

That said, you live in an idealistic world. You’re more advanced than 99% of people who are looking to build an online business.

Do I think split testing is wrong for everyone?

No.

I just think it’s wrong for people who have more important things to do. Like…

…actually driving traffic.

…writing high quality content.

…promoting that high quality content.

…getting a business model in place.

And more.

I filmed this video for one reason:

People write in to me all the time and say, “Derek I have a conversion problem. I keep running split tests and they’re not working.”

When I dig deeper I find out they have no traffic and they’re wasting their time, spinning their wheels in the mud, trying to “lotto” their way to success with a successful split test.

And that’s why I say that when people are just getting started, with no revenue streams, no base level of traffic, no nothing, split testing is a waste of time for them.

And yet…

People are STILL split testing. Like many other people above who commented.

Why are they split testing?

Because they’ve been duped. They were told how easy it was to run a successful split test. They saw the amazing results published on split test sites where changing button colors skyrocketed conversions. And they think, “This is real easy. I should do it.”

And they do do it. At the expense of the rest of their business. They turn into dopamine seeking slot machine pullers awaiting a successful test result instead of working on the fundamentals of their business.

And that’s a big mistake.

That said…

You mentioned:

“In fact, if you have a low traffic website, and you actually know what you’re doing ”

This idea is fundamentally flawed. I’ll tell you why:

The people who I see running split tests don’t even have a product yet. They don’t even have visitors yet. And they most SURELY do not know how to run a proper split test.

You say “you actually know what you’re doing,” as if everyone who starts running split tests knows what they’re doing. They most definitely do not.

You know what you’er doing. They don’t.

Now if they’re at the point in their business where they can afford to higher a professional conversion optimization person, I most definitely think they SHOULD hire one. As a matter of fact, they should hire you ;-).

But remember they don’t even have a revenue stream yet. And that’s why I say it’s a big waste of time.

You also mention this:

“I started my blog (ConversionXL) just 2.5 years ago. I wrote all the content myself, did all the marketing myself – typical one man show. I got to 100,000 monthly readers in 12 months WHILE running split tests to gain insights about my audience. It’s not this vs that at all.”

Again, you’re much more advanced than most people who are looking to start building their online business.

I know how to run split tests, and I chose not to. Because my time was better spent elsewhere and I wanted to free up headspace to focus on doing what I do best… getting readers.

And like you, Social Triggers grew pretty quickly. I know you know that, though.

Keeping this all in mind, I should say one more thing:

If you’re at the point in your business where you can afford to hire a professional, you absolutely should do it. But if you’er not a split testing master, there’s no reason trying to become one when you still have other more important things to work on… like product, business model, and other business fundamentals.

Reply

ScrewtheSystemJoe

Think Derek makes a good point here, for the small time blogger looking to blow up, it’s all about use of time.
Split testing is like the icing on the cake and if you haven’t got much cake, there’s little point in tasting it!

Reply

DigyTouch

Unless you are a big brand you shouldn’t spend lots of time on split testing. Instead I would use this time for growing traffic from social media. As Derek said, there are better ways to make sign-ups higher. Grow your targeted traffic!

Reply

Rich Page

Ah good old Derek. As controversial as ever with your article titles ;)

Coming from another ‘conversion guy’ here… I disagree with some of this.

While I agree split testing is hard to get great results all the time, and people run sucky tests much of the time, I have to disagree with a major point in this.

It’s often VERY EXPENSIVE OR DIFFICULT TO DRIVE NEW TRAFFIC as a way of increasing sales or leads. Many online business quickly hit a glass ceiling for traffic they can’t break through easily or afford to spend more on. I hear this from my clients all the time. And its just plain not as profitable as fixing major site conversion issues….

“Doubling your conversion rate means you halve your CPA (cost per acquisition)”. That’s a fact from conversion-rate-experts.com. And WHO doesn’t want to do that?

That’s why sometimes DOING SPLIT TESTING IS MUCH MORE COST EFFECTIVE than wasting money driving traffic to a ‘leaky bucket’ of a website, that doesn’t engage your visitors or meet their needs properly.

It’s also one of the 3 major reasons why online business fail.

Rich Page: Website Optimizer

Reply

Natalie Sisson

Definitely a hot topic right now. I’m all over optimizing in general in the areas that I consider high priority for my online platform that lead to the results I want.

What I would want to split test though is the ‘sexy video’ segment of you working the camera, vs you looking away ;)

Reply

Paul

I think people don’t understand split testing. Changing a ‘buy now’ button is like saying changing the seat covers on a sports car to make it faster. You saying ‘money points’ just confuses it further. You’re isolating a part of split testing.

Split testing should mean testing offers, prices, copy, testimonials, adding video content. Why not make your offer more of a no-brainer? Or the position of an image, captions. Look at companies like Agora and what they do.

I definately wouldn’t recommend shunning response optimisation for the sake of traffic generation. You really need to go for both.

But I figure it depends where your talents lie, and how experienced you are.

Reply

Paul

Derek, I think you personally could split test your email copy. Especially the auto responder stuff.

It reads a bit too long. Shorten it a little.

Your blog posts are spot on.

Reply

Luke Thomas

I do agree that split testing with small amounts of traffic is not a good use of your time, however, the overall message you convey (whether you wanted to or not) is that A/B testing is a waste of time.

That’s wrong.

First, if you setup A/B tests based on YOUR intuition (button colors, etc), you’re definitely wasting your time. A/B tests should be a result of some type of qualitative feedback from customers. This will improve your “batting rate” significantly and cut down “all this time spent setting up A/B tests.”

Secondly, I haven’t seen a test (if run correctly) “regress back to the mean.” I’d love if you provided specific examples – I’ve worked for 2 companies with absurd amounts of traffic, and have yet to see this regression (and I run A/B tests on a daily basis…p.s. – I’ve also paid my salary back several times over.) Clearly not everyone can run tests w/ millions of visitors, but a/b testing is certainly not a waste as you suggest.

Thirdly, you portray gaining more visitors and split testing something that’s mutually exclusive..which is not the case. This can be done at the same time.

I think you had good intentions with this video, but unfortunately you’re sending the wrong message.

Reply

Bhaskar

That’s a good point that not only traffic, but a solid hypothesis is what an effective A/B split test makes. You need both, not one or the other.

Reply

Beth Anne

Great advice here, Derek! It’s the same principle that you’d apply to a psychological study – the results have to be statistically significant to matter and usually that means having large test and control groups. I’m remembering p<.05 but don't ask me to do the statistical analysis!

So my answer to the question – have you ever done something because everyone else was doing it….?
Link parties! For Mom bloggers this is supposed to be the holy grail of getting new traffic to your blog. In reality, the link party has to be HUGE in order to get you any added readership on your site. I spent a few weeks killing myself trying to submit to every link party on the planet only to get 2 hits from each one. The big link parties can really be worth it, but most are an absolute waste of time, and I wish Mom bloggers would quit touting them as a "great way" to grow your blog.

Reply

Erin

One of my yoga teachers frequently says, “Just because you can, doesn’t mean you should.” She is referring to practicing yoga poses, not split testing, but it’s the same idea. Even if your body will bend into a specific shape, it’s not necessarily good for you/sustainable/necessary for any functional life purpose. There is risk for injury, waste of energy, waste of time when you could be practicing a more beneficial pose, etc. I love this teaching and I share it with my own students as well. Thanks :)

Reply

Jamie

This was cool, I felt like this split testing is something I needed to do right out of the gate. I see how that might not be the best use of time. I’ve only been up a few months now so will put that on the backburner but it definitely interests me.
-Jamie

Reply

Jan

I hate split-testing so much. It makes my teeth hurt. This video just validated my feelings about it!

Reply

David Cunningham

Amen! Amen! Amen Derek! This is one of the smartest videos I have ever seen. Your views on split testing is right on the money, it is what I do on my blog Up Next. Thanks for a great video Derek!

Reply

AJ & Serenity Services

Hey Derek great advice! I certainly agree that split testing can take up so much time and that there is a time and place for it.

There are so many aspects to running an online business or profitable website (copywriting, branding yourself, promoting yourself, etc.). As online entrepreneurs, we should really take the time to do an 80/20 analysis on our activities and figure out which 20% of our activities yields 80% of the results we’re looking for.

One thing I’ve heard people talking about is setting up your business entity just right for the sake of reducing your tax bill. Just because other entrepreneurs are setting up corporations or LLCs doesn’t mean you have to. There are so many factors that go into this. At the same time, something like this should be talked over with an accountant, attorney, or other expert.

Reply

Julie Brett

Derek… I’m two minutes into the video and I still have no idea what a split test is…

Reply

Julie Brett

I also have no idea what a conversion is…
I remember watching a video you did about not using big words… how about jargon? Is that ok? :P
(I love you videos usually. Just sayin’… <3)

Reply

Tim

Chasing too many different industries at once… committing to too many different traffic strategies…. both examples of failed attempts because I did something I *could* instead of focusing on what I *should*. I find a lot of service providers, including consultants / experts are particularly guilty of this (myself included).

Reply

Kamila Gornia

Thanks for keeping it real, Derek. I love this video. It seems like I see split testing info allll over the place and it’s overwhelming. Glad to see it’s okay not to focus on it and not feel like I have to either.

Great tip to focus on the money point too rather than the entire website. Definitely much more manageable.

Have a good Memorial Day weekend! :)

Reply

Javier

That’s a bad ass shirt, Derek!

Reply

Joe

I take this philosophy to the extreme… I never split test anything at all frankly. I know thats a huge mistake as a marketer, however, it’s served me well over the years. I spend all of my time focusing on the best content that I can possibly create and let the cards fall where they may. Is that hurting me? Maybe, maybe not. But at the end of the day, I’m happy, my customers are happy AND my business is thriving.

Reply

Michelle Marie McGrath

hi there, this is really interesting Derek. I wanted to change my opt-in gift on my site and enlisted the help of a developer to do it for me. He suggested they do some split testing with my current opt-in and my new proposed one. I already strongly felt my new one would be much more effective (without testing). I should have just listened to myself but went ahead with the testing thinking that he knew better than me. 6 weeks later and lots of time wasting (and more money spent) he came to the conclusion that yes my new proposed one would be more effective…. er thanks…

Reply

David Kadavy

You beat me to it! I teach design to entrepreneurs, and I have to shake my head when people ask me questions like “what button color is the best for conversions?” They’d be better off spending that energy on nailing their customer and product.

Volume is an issue, too. Sure, Google can test 43 shades of blue, because they’re Google. A .1% change in conversions could be worth millions of dollars.

Reply

April Greer

Hi Derek,

A regular here on Social Triggers, just a couple of comments on split testing your wardrobe:

- You look really pale and washed out in this video, and I think it’s the shirt color. I’d pick more colorful shirts to contrast your skin.

- I think you look more “hands on” and “down to business” with your shirt sleeves rolled up. The long sleeves all the way down make you look like the guy that doesn’t normally get in front of the camera.

Just my 2 cents.

April

Reply

Norma Maxwell

Thank you Derek! I have had several small business clients over the years starting out with limited resources, and I will definitely be pointing them to this video when they start stressing (as they often tend to) about the little things when they really need to be focusing on putting their best out there to get themselves established (making some money!)–then they can take the time to split-test and obsess over metrics all they want! Great advice–thank you! ~N

Reply

Louis Blythe

Interesting points here Derek!

Mind you I do see the value in split testing structural changes.

Louis

Reply

Natee

This is useful. I think the big idea of this post is that we should put first thing first. Thanks Derek. Just because you can doesn’t mean you should. I love it!

Reply

Alexey Puznyak

Derek, you are absolutely right. From my experience, you definitely NEED split tests when:
a) you have build up the scale of your business and already have stabilised your processes (i.e. you can predict how much traffic do you need in order to achieve what you would like to achieve and you know where to take it)
b) you have squeesed as much as you can from the conversion funnel by other interments (monitoring, getting back to the customer via mail, phone, chat whatever else).

My experience – we have spend much more time expanding referrals network than running split tests. As a results our traffic grew from 35 000 a month to 1,4 mlm a month.

On this scale, you have two significant benefits:
a) 1% increase in conversion means HALF MILLION dollars in additional revenue a year in a business I manage
b) you can run multiple tests at the same time and you don’t need weeks in order to get at least some statistically valid results.

In other words, if you are in the very beginning of your journey – don’t waste your time with split test. The are many much more effective ways of increasing your results than split testing.

Reply

Ahmed Aljonaid

I first started using split-testing 19 years ago when I first started doing business offline. It was a necessity back then when we had to send 20,000+ postcards of a sales letter to find out what works and what doesn’t. But then

I started utilizing the internet since 1998. And yes, we started split-testing right away, and let me tell you, it was hard back then with almost non-existing tools to do such things.

Derek, you are right. It was a waste of time, effort and money..since time is money. We didn’t have the enough traffic to apply those tests on. Instead we should have focused on increasing traffic and maintain conversion rate.
Things like that are the very same reason why I decided to start online marketing again from scratch and unlearn everything I’ve learnt in the past.

Thank you Derek for brining that up.

All the best.

Reply

Patty Ann

Derek ~ there is no such thing as a mistake. Only lessons learned. :)

Reply

Joy Healey

Thank goodness for this common-sense. I always felt bad because I didn’t see the point of split-testing for MY business – yet. But I didn’t like to admit it. I shall focus on traffic generation with a clear conscience.

Joy

Reply

LeslieZ

Derek, This makes so much sense and thank you for saying it. I’m getting to the point in my business where I am being told I should be doing split testing and I was dreading it.

But now that you’ve open my eyes to what makes of why to split test, it doesn’t seem so overwhelming.

LeslieZ

Reply

Laura Lynn

Derek,

First I have to say, I loved the “sexy” part of your video! ;)

What you said makes so much sense that I wondered why I didn’t think of it before. I’m still building my site, but I’m learning to KISS and not put too much “busy work” on my plate, but concentrate on what will benefit my audience and bring them to my site.

As to what I *almost* did just because I could… but thankfully decided that “just because I can, doesn’t mean I should”. I was creating the intro to my craft class videos in Powerpoint. I started goofing around with all the ways to animate the photos and titles in that short video, when my other half mentioned that I could waste a LOT of time on little “artistic touches” like that. I realized he was right, and within 30 seconds decided upon a “theme” and created a simple .jpg intro for the video.

Reply

Cem Goknil

This video and the 300$ haircut video reminded me of an HBR article I read a while back; “Three Rules for Making a Company Truly Great.” (Can be found on Google – didn’t want to put up links here to not be disrespectful to st.)

Derek’s approach to split testing echoes rule #2:
2. Revenue before cost—that is, prioritize increasing revenue over reducing costs.

And the 300$ haircut definitely reflected rule #1:
1. Better before cheaper—in other words, compete on differentiators other than price.

Both of these rules really helped me out financially while I was struggling and switching from being a webdesigner to an agency (basically a business owner) in my early 30s.

And there was one more thing that helped me out: As a small business or startup, we cannot sell for cheap & build in bulk. We are 1 or 2 people max: That means we can build a few things at most in a month. Big companies with production lines & logistics divisions can build in bulk, but small companies just can’t. That means, we have to position ourselves at high price & unique&great service.

Reply

JasonS

“Just because you can, doesn’t mean you should”…. Thank you for the quote.. I have that stuck on my wall..

Reply

Leave a Comment

Previous post:

Next post: