🔬Is A/B Testing Useful?
How to YouTube #6
Hey YouTubers,
Back in February this year, I had a call with someone from the YouTube channel Charisma on Command.
The purpose of the call was to better understand A/B testing because apparently they did lots of it. It turns out, they do more of it than any other channel I know. I’ll show you what I mean in a minute, and why they do it, but first I’ll do a quick explainer.
What is A/B testing?
It’s comparing sample A against sample B with the same audience and seeing what they prefer.
For example, running 2 different FaceBook ads, and seeing which one converts more customers to buying the product.
On YouTube, A/B testing generally refers to seeing which title or thumbnail leads to the most views for a given YouTube video (even that has some nuances though, which I’ll get into).
Why is it useful?
As most of you know, titles and thumbnails are incredibly important. They can literally make or break your video.
If you can’t get someone to click on your video, you can’t get a view, no matter how good your content is.
This is my YouTube homepage right now.
The only things I can see are titles and thumbnails.
It doesn’t matter how good the content is in any of these videos at this moment in time.
But you’re smart, you know this already. Thumbnails are important.
Therefore, A/B testing can be incredibly powerful because you can dramatically change the number of views a video can get by changing the thumbnail.
Here’s an example from Ali’s channel. I’m using the CreatorML chrome extension to show this data. You can see it too just by clicking on any YouTube video with this extension installed.
The left is an uninspired unoriginal thumbnail that was made in a rush (if you’re reading on your phone you might need to zoom in a bit).
So, we ran an A/B test with the thumbnail pictured on the bottom right. It turned out that the Click-Through Rate on the one on the right was much higher, so we changed the thumbnail. The views took off and the video now has almost 1m views.
But look at the Charisma on Command channel:
Every dot on that graph in the top of the image is an A/B test in action. Basically every video they post looks like this.
Their approach to YouTube is to only post a video every 2-3 weeks and make sure they haven’t wasted all their effort making their video by messing up the thumbnail.
But why don’t you just change the thumbnail outright?
Great question. Mainly because we’re often not good judges of what thumbnail will perform better. Data is a more accurate, but not perfect, indicator of the better title and thumbnail.
How do you do it?
For the last few years most people have been using Tubebuddy to run their A/B tests. You just give them access to your channel (this is fine don’t worry), then select the video you want to run the test on, upload an alternative thumbnail and choose the duration for the test.
They then swap out the thumbnail every 24 hours for however long you’ve chosen, and analyse if viewers click more or less, and watch more or less, and show you the results.
More recently people have been using thumbnailtest.com which we now use and I can highly recommend. It was built by the legend Rox. It has more options than Tubebuddy (like hourly A/B testing) and is much much nicer to use.
But even more recently, YouTube have announced that they are releasing an A/B testing feature in YouTube studio just for thumbnails. We have early access, it looks like this.
The reason this is better is because YouTube shows different thumbnails to different viewers at the same time, rather than swapping the thumbnail every hour or every day which is what Tubebuddy or Thumbnailtest.com do. As a result the data is more reliable.
They also give you the results in watch time, not based on Click-Through Rate, otherwise this would encouarge clickbait thumbails. Also, the thing YouTube ultimately cares about is people watching videos they want to watch.
What does this all mean and what should I do?
Here’s a list of things to think about:
If your channel is regularly getting over 10,000 views per video, and you have enough money to buy a subscription to thumbnailtest.com and the time to make more than 1 thumbnail per video, I’d start A/B testing on every video you upload (and have uploaded, if you can).
If you’re getting fewer views than that per video, I would just change the title or thumbnail outright, and not bother with A/B testing. You can ignore this entire email :)
Either way, make thumbnails that are intriguing to increase Click-Through Rate, but also are highly aligned with the content of the video to increase Average View Duration (and therefore total watch time of the video overall).
Generally, you can do A/B tests at any time and for any duration. The longer they are the more reliable they are likely to be, but also bare in mind that Click-Through Rate will change day by day anyway because of natural variations, so the results are never perfect.
If you think a video is underperforming, I recommend changing titles or thumbnails outright within the first week, but after that setting up an A/B test.
If you do want to run an A/B test within the first week, I’d run hourly tests for 12 hours. If you want to run a thumbnail test after the first week, I’d do it daily and set it for 10 days total.
A/B testing is not a perfect science. All viewers are not created equal and have different behaviours on different days. It could be that you think one is better based on the results but actually the other is better in the long term. A/B testing should be a factor in your decision making.
Wow that was a long one.
There’s actually so much to say on this and every channel should approach it differently. I’ll try to write more on this in a future issue.
If you have a question, feel free to comment below or reply and I’ll do my best to answer :)
Have an epic week!
Tintin 🧑💻
The YouTube Tin
🔬 My Channel: I make videos (really just screen recordings) about YouTube to help you guys as much as I can and share more of what I learn at my job. Let me know what you’d like me to make videos about. I’ll share as much as possible!
🎙️ Podcast: How to Make More Out of Brand Deals with Justin Moore and Jay Clouse
🍿 Video: This video from Elizabeth Filips has gone crazy compared to her most recent videos. What’s your take on why that is?
🪓 Tool: I’ve been using WordCounter to track my writing each day so that I can hit my daily goal of 300 words. YouTubers can use it to set goals for planning videos or writing scripts.