Why This Entrepreneur Assumes Nothing and A/B Tests Everything

An A/B test (or split test) is a common method used to improve your website, ads, or any other digital property where you can measure performance. You simply pit variant A against variant B (all other conditions being equal), and see which one performs better.

Instead of assuming what color, copy, or layout might work better, it's a data-driven way of choosing which changes you stick with and which ones you scrap.

Megan Bush is the co-founder of Kopari Beauty, 100% organic coconut beauty products, and attributes a large part of the company's growth to relentlessly conducting these tests.

On this episode of Shopify Masters, you’ll learn how she goes about A/B testing every aspect of her site. 

Listen to Shopify Masters below…

Download this episode on Google Play, iTunes or here!

 
“A lot of times you get quicker results if you really just change the entire thing up—even change the product you’re showing.”

Tune in to learn

  • How to take advantage of and outlive a trend
  • The advantages of buying ads in traditional media
  • How to work backwards to test an ad.

Show Notes

Transcript:

Felix: Today I’m joined by by Megan Bush from Kopari Beauty. Kopari Beauty is passionate about products that make you look gorgeous with ingredients that make you feel great. It started in 2015 and based out of San Diego, California. Welcome, Megan.

Megan: Thanks, Felix.

Felix: Yes. Tell us a little bit more about Kopari Beauty and some of the popular products that you sell.

Megan: Absolutely. Kopari Beauty sells coconut oil based beauty products, and our hero product is our coconut melt, which is 100% organic coconut oil. For those of you who’ve heard about coconut oil, it’s got amazing multi-tasking properties, both for health and beauty, and what we’ve really done with all of our products at Kopari is take coconut oil to the next level by adding all natural ingredients that really just boost those benefits of coconut oil for beauty purposes.

Felix: Awesome. Where did this idea come from? How did you and the team start going down this avenue?

Megan: Actually it’s kind of a funny story. Kopari has four founders, and the founders basically all came up with a similar concept for a coconut oil beauty company and came to a common friend with the same idea within like a week of each other. That common friend introduced everyone to each other and they said, “You know what, this is so weird that we all had this idea at the same time. Why don’t we all join forces and launch a brand out of it?”

Felix: Yeah. That’s funny that all of the founders came up with the idea at the same time. Once they all met and decided that this was a great idea, we all came up with this idea, what were the first steps? How did you even begin to take this idea … Maybe we’ll start there. What was the idea itself? Was it just to create, if you can explain to me, what was the idea, when they all came together, what was the idea that they wanted to pursue?

Megan: They definitely all wanted to use coconut oil to some degree and use it into different beauty products. One of the founders wanted to create hair care, one of them wanted to create skincare. They all had kind of thought they would … Coconut oil is so amazing both for hair care and skin care and body care. The idea was to start with one product and really just ramp up with more product lines as the business got rolling. That’s why it’s confusing as to who really came up with the idea or how did we really start with this, but once the idea was brought up by multiple people and everyone met each other and decided this is something we really wanted to start a business on, then it became time to really just start looking out for some beauty labs, actually.

They interviewed a bunch of different beauty labs in the Southern California area, found one that they felt really comfortable with and they started just the product development at that point. Very quickly, while the product was being developed, they had a lot of conversation about, “How are we really going to launch this brand?” They definitely talked a lot about whether they were going to go into distribution through salons and spas, whether they were going to go straight into stores, or whether they were just going to focus online.

At that point is when they decided that they really wanted to test the market in the direct consumer channel, that’s when I got brought into the company. They gave me a call, I had worked with one of the founders years ago on a couple different brands that he had. I was working for an e-commerce agency. He called me up and said, “You got to come back. We need your help. We need you to help us sell coconut oil based beauty online.”

Felix: I’m assuming you already had a stable job and everything. Someone calls you up and says, “We want you to join a new company, a startup.” What made you attracted to the opportunity?

Megan: Well, I was working for an e-commerce agency which I absolutely loved, but when the opportunity fell in my lap, there was two things that really drove me to take the job. Number one, the guy who called me, James Brennan, I worked for him for a few different companies that he’s had. He’s always had a knack for spying trends and really doing well with the businesses that he starts. I’ve been lucky enough to see him be successful and work with him in a lot of those successes. Number one, the fact that he was putting this together, I thought that’s a great brand that I should be a part of just due to the fact that he’s involved himself with it.

Number two, working for different agencies, I love digital marketing, I’ve been doing it for my entire career, and I really wanted to work in house for one brand to get a full-circle view of what the digital marketing really entails. When working for agencies, you’re always only seeing a piece of the business and never full-circle exactly what’s going on as far as even just inventory management and fulfillment services and how these different offline channels really effect the online sales. That was really attractive to me. I thought that’s definitely something that I could learn from and so that was ultimately why I decided to join the company.

Felix: You were mentioning before, all of the founders wanted to do something with coconut oil. You saw that these founders that were starting the businesses had eye and understood or ready to take on opportunities in the marketplace. Talk to us a little more about this. What did they see in the market that made them decide that this was the direction to go in? What did they see about coconut oil specifically?

Megan: Coconut oil, it’s been a very buzz-worthy trend for the past couple of years. I had personally, when they called me up a couple of years ago, I had personally already heard of it and I had been using it just as a body moisturizer. I’d use it for something called oil pulling. It’s this amazing ingredient that you can use for cooking, you can use it for your hair, your skin, your body, etc. There’s a lot of buzz around it especially in the healthy and beauty space. If you follow any blogs or any influencers in social media, everyone seems to be talking about it. I think what they found was a void in a space because a lot of people … You can buy a jar of coconut oil in a grocery store.

Basically what they, and all the founders were experiencing and I think a lot of people out there were experiencing, that you go and you buy this jar of coconut oil on the grocery store shelves, you take it home, you start cooking with it. Then you realize what an amazing moisturizer it is and you take it into your bathroom and you use it as a moisturizer. Then you take it and you oil pull with it and then you take it back to the kitchen and you cook with it. It starts to get a little bit confusing what this product is actually for. Is it a beauty product, is it a health product? What are you actually doing with this?

A lot of people had been focused on the health aspects of coconut oil, and it seems as though there was a void in the market to really talk about the actual beauty aspects of coconut oil and how can we take this amazing ingredient and actually make it easier to use as a body moisturizer and make it easier to use as a hair mask and make it easier to use as a flyaway tamer and as a makeup remover, and all these amazing uses that you can do with coconut oil. How can we really amplify those benefits by incorporating complementary ingredients and also just make it a much more enjoyable experience and put it in beautiful packaging and really buy into this tropical lifestyle that we’ve created?

Felix: That makes sense that they recognized that there was a lot of talk about the health aspects, but then there was this gap where there wasn’t that much focus on the beauty angle of coconut oil. How did they know that it was the beauty angle that was going to be the one that was the best business to build on?

Megan: I think just the pure make up of coconut oil in general. It’s not just a product that makes you beautiful. Its core, it has this lauric acid which is something that’s found in almost no ingredient in nature. If you just study the chemistry of the coconut oil molecules, it’s something that just doesn’t really exist. I think they realized that there’s something unique about this. Oils are very popular in beauty right now. Coconut oil is just something that’s just, it’s so universal. Everybody’s heard of coconut. Everybody recognizes what coconut is, whether it be the smell or the taste or whatever. They knew that it had mass potential to really go out and get the attention of a lot of people.

I think that for women, having something that’s multitasking and something that they’re able to use for multiple uses is so beneficial. If you look at most girls’ beauty counters, they have so many products. To have something that’s sitting on that counter of your bathroom that’s not only beautiful in the jar but also something that can be applied to multiple areas, like your hair, your face, your body, etc., it just makes so much sense. It simplifies things. It makes your life a little bit easier, especially when you’re traveling and you want to throw something in your bag. You don’t have to pack all these different products for all these different uses, you can take this one core product and use it for so many different things. I think that was really attractive to them.

Felix: Now do you think it’s possible to reverse engineer this process that the founders went through where they’re able to identify a rising trend and then finding a new way to package it and bring it to market?

Megan: That’s a great question. Reverse engineering the process. I would say that, again, I wish I could give more insight into that. I think you have to just pay attention, as simple as that sounds. I think paying attention and really recognizing when there’s … If you hear something repeatably, you’re paying attention to what the bloggers are saying, to what the influencers are saying, and then you’re having … I wish I could answer it better, but I’m not really sure I would do that.

Felix: Yeah, I mean I think that makes sense that you want to pay attention to what the influencers, the blogs, what they’re talking about. Now, did the founders themselves, did they have experience in the beauty space? How much experience did they have to create a product like this?

Megan: One of the founders and the CEO, Bryce Goldman, he actually was born and raised into the beauty space. He has a family beauty distribution store with spas and salons in them. He’s been familiar with beauty products. I think that’s been really helpful because he’s been watching coconut oil and different trends within the beauty space for years and waiting for someone to come out with this really great product that would highlight coconut oil. When nothing came, I think that’s where he really saw his opportunity to build a brand around it.

Felix: How do you manage a product, a brand like this, that is a trendy product? Like you were mentioning before, it become very popular in more recent years and it only seems to be getting more and more popular. Is there a point where it’s going to taper off? How do you manage that potential outcome?

Megan: I think that’s a really great question and it’s certainly something that we’ve discussed internally. I truly believe that if a product has the efficacy, if it works, if it’s just a very good product and a good brand, I think it outlives all of the hype and buzz that’s generated around it. That’s another reason why we’re so focused on taking this ingredient and adding other ingredients that definitely improve the efficacy of it and the performance of it so that it’s not just one ingredient and it’s not just a fad or a hype. It’s something that’s going to live on forever because it truly is that amazing. It’s not something that is ever going to go away. It’s actually been used for beauty and health for centuries in different cultures. I think it’s just recently becoming popular in the United States, but I actually think it’s been quite popular in Asia for a long time.

Felix: Yeah, I think this is a great example of using a trend to essentially launch a product and to get that visibility, but at the end of the day you still need to have a product that is going to work. Like you were saying, needs to have efficacy, needs to actually be a product that has value. You can’t just ride the trend and be empty once people use the product. It actually have to be effective. I think that’s a great example of, well this is a great example of a product that it has been able to take advantage of a trend but then of course deliver the value at the end of the day so that you don’t die with the trend if it ever does taper off. You mentioned that one of the first steps that the founders had to take was to identify and start working with a beauty lab. Talk to us a little more about this. What is a beauty lab?

Megan: Basically, they’re the ones coming up with all of the formulas and making sure that they’re safe. They know what ingredients to be using to create the products that you’re looking to create. We started speaking with a few different beauty labs. It’s essentially an interview process, just like with any vendor you have. You speak with the team and you determine who understands the vision that you’re trying to set forth and who can come up with the best concepts for your team.

Felix: How does the interview process work? How did the team identify which beauty lab to go with?

Megan: I wasn’t a part of that process. I actually came on after they had already selected the main beauty lab that they wanted to work with. I know that they definitely sat down and met with multiple labs. I think that being in Southern California is a blessing because there are multiple labs that we were able to drive up and meet with and sit down with and discuss. I think it’s a matter of not only who you feel really understands the vision of the brand and of the company, but also someone who is willing to take risks and push limits as far as what they want to do chemically speaking to the product.

Felix: At the end of the day, what does the beauty lab deliver to its client?

Megan: When we’re looking to launch new products or when we’re working on formulating new products, we’ll work with them to actually create those formulations and to actually piece together what ingredients need to go into that product. Then once we decide on, “Okay, this product is done,” and we completed the process and we sign off on the end product, then they’ll actually go in and actually manufacture the product for us.

Felix: Very cool. Are they able to hand hold a client through this process or do you need to have some kind of background, or expertise, in beauty before working with a beauty lab?

Megan: That’s a really great question. I think that one of the perks of working with at least the lab that we’re working with is that their main focus is to actually create the product and make sure that it’s safe and it works for everyone. Our main focus is to really take their creation, or the creation that we worked with them to build, and market it. They really want to focus on they want to take the heavy lifting out of it and make sure that the product does what it’s supposed to do and works well and has a long shelf life and do all of that homework while our focus is to really how are we going to communicate this to our customers and how are we going to get this product in our customers’ hands and how are we going to make sure that it’s what they want and they love it?

Felix: Awesome. When they called you in and you started to work with the company, what did you start working on immediately? What was your main focus?

Megan: The first week I started, it was far before we actually had product in stock to sell. I knew coming in that I wanted to make sure the site was built on Shopify. That was the quickest and easiest decision that I made, just because working for other e-commerce companies I’ve seen the pain of dealing with a lot of the backend stuff that you just don’t have to deal with on Shopify. I didn’t really want to deal with PCI compliance. I didn’t really want to deal with hosting or any of that headache. I really just wanted to focus on marketing the product.

I had done enough research to know that that was going to be really easy just to do on Shopify, so that was a quick and easy decision. It’s kind of embarrassing, but I spent my first few days at work going through the entire Shopify app store at the time and not only using that as a tool to decide, “Okay, what marketing channel should we really launch in, what’s out there, what is there a pre-built integration for so that I don’t have to go out and try to find a super expensive solution for it?” That was my first order of business was to go out and basically build a marketing plan that really just focused on what already existed within the Shopify community.

Felix: Yeah, I think this is a process that a lot of new Shopify owners go through is this to identify which apps, what kind of plug-ins they should tack onto their site. Did you choose any apps right away? Do you remember which ones you chose? I think this is a point that maybe some entrepreneurs get stuck at, which is what do they need to get started?

Megan: Yeah. I think a couple of the apps that I chose right away, Yotpo for reviews. I saw so many stores were using it. I was familiar with the tool myself so it seemed like if so many stores are already using it, maybe there’s something here so let’s not reinvent the wheel. We definitely used Yotpo. We used AdRoll for remarketing initially. Definitely Google Analytics, I’m definitely an analytics junkie and having a free tool like Google Analytics is so helpful, so that was one of the tools we used. Initially though, I don’t think we still use a ton of the tools that we used at the very beginning. There was an app called Fomo.

I think it had a different name at the time, but now it’s called Fomo where it’s basically an order alert system so that there’s a little pop-up that says, “So and so purchased your product in Oregon today,” or something like that. I remember thinking at the very beginning, being a brand new beauty company there’s so much competition in the beauty space and how are we really going to stand out? Once we create these amazing products and once we start our process for advertising and everything, I wanted to make sure our store was a place that people would come to and trust.

I wanted to build that trust through a variety of ways, but I wanted the store to look very professionally done. I didn’t want it to look like something we just threw up quickly or haphazardly. I wanted to make sure that customers knew that this was legitimate. Using that app, Fomo, was really helpful in the initial stages because people were able to see that other people were actually buying the product and that really provided this social proof that that’s a legitimate site.

Felix: Yeah, I think that that’s a great idea too to provide that social proof, to show that other people are trusting this brand and by having that, by having reviews, like you mentioned before, installing software for the review system, having all of them gives people that trust to trust in a new brand, trust in a new site. Can you talk too a little bit more about this? What other changes or what kind of conscious decisions did you make with the actual site itself to build trust when you have a brand new store or brand new product?

Megan: Beyond just the reviews and the order notification app, I also made it a huge priority to immediately start AB testing on the store. That’s actually where a bulk of my experience has come from, was conversion rate optimization. I knew that going into this, with a brand new product, with brand new packaging, it’s just brand new and I knew that it was going to be very difficult to determine what exactly about our product is going to resonate with people. Immediately I actually signed up for an Optimizely account and started AB testing on the website. One of the first things we actually ran was testing product descriptions. We had spent a bunch of time really focusing on our brand and our brand voice and how we wanted to describe these products.

Ultimately, we received so much feedback from just friends and family that were concerned about the way we were explaining the products and talking about them. They said, “Your language is a little too fun. It’s not quite functional. You need to be more descriptive about what the product actually does.” We obviously took that feedback and we didn’t necessarily shift gears, but what we did was we started to create more functional language and then AB tested the product descriptions on the website to really figure out what was resonating with people. What we found is that people actually responded really well to the more fun branded content than they did the very literal, “Here’s how you use this product,” information.

Felix: I think that’s a great lesson to be learned that when people, especially people that aren’t necessarily your customers, maybe friends and family they have your best interests at heart, but they’re not actually, a lot of times, your target customers. They’ll come along, of course everyone wants to help, and give their advice on how they want you, not how they want you to, but how they suggest you message the product, write the product descriptions. You guys didn’t just take that information and run with it. You tested it out to see if it was actually going to work or not.

I think that’s a great insight. Now you mentioned with AB testing, I think this is something that a lot of entrepreneurs, a lot of store owners want to start doing, but you need a good amount of traffic though, right, to get enough data to make these decisions or run these tests to make these decisions. Were you already getting a good amount of organic traffic already or was it through paid traffic to get the influx of data to begin AB testing?

Megan: We definitely were running paid traffic. I think most store owners can relate to the fact that you launch a store and you can’t just sit there and wait. It very rarely picks up on its own unless you’ve started to build the buzz through a Kickstarter or something else. We didn’t really have that. We didn’t have any huge piece of press that came out immediately. We did do some sampling initially, but we always knew that doing paid media was going to be very important to getting our brand out there and to getting eyes on us.

We also made traditional PR a very high priority in the beauty space. I think that it’s very important that people see your brand in traditional publications as well as online publications, just to give it that credibility. We also did work with influencers. I think between the sampling, the PR, the influencers and the paid media, which is a lot, that’s where we started to really see the brand pick up speed and the traffic pick up and that’s when we were able to really test the market like that.

Felix: Yeah, one thing you had mentioned to me, I think offline in an email, was about how you were trying so many different things and you weren’t exactly sure why things weren’t picking up right away until it all came together through all these efforts over time, adding up to a point where people started recognizing the brand from multiple touch points. Talk to us a little bit more about this. How did that come about? How did that realization come about that you need to hit them on all these different kind of channels?

Megan: I think initially we had, as with any product, you hear all these different stories about, “Oh, we just really focused on influencers. Oh, we just really focused on traditional PR.” Everyone has their niche channel. We started slowly with a bunch of different channels and nothing really took off on its own. While we were getting a few traditional PR pieces and we were going out to influencers and we would pay them to post, we never really saw a big influx. Nothing really happened. Then we started running paid advertising. Again, things just started very slowly. I think one of the challenges with the paid advertisement is that we were going and working with people who are highly experienced in doing paid advertising.

Again, just like those friends and family who said that maybe our product descriptions weren’t good enough, we were experiencing the same struggles with these very experienced paid advertisers where they were saying, “You know, your product just doesn’t resonate. You need to be more literal. You need to be more specific. Your product’s too confusing.” The feedback, basically, was always blamed on this product. What I learned was that when I looked at the ads that they were running it was very much talking about the free shipping opportunity and the best deal and all these different benefits to the products that just weren’t really leading with what it was, which is coconut oil is amazing for you.

We’ve built an amazing brand around it. We have amazing imagery that goes with that. We know with our product that they’re in beautiful packaging. I feel like they didn’t trust the brand message and the brand imagery enough to lead with that. They were leading with the free shipping message, which is best practice but it just wasn’t working for us. Once I heard a few too many times how our product just wasn’t really going to be able to be sold online and we needed to try to do it in stores, I just refused to believe it. We brought the paid advertising in house. I basically spent day and night setting up ads and changing up the audiences and switching up the content and really testing the messaging that way.

Over time, week over week, I saw it slowly start to ramp up. In the meantime we were also testing very intermittently different influencers, we were getting very small press pieces. Then all of a sudden we just saw this spiral effect where all of the efforts started rolling together and people started commenting on the paid advertisement that, “Oh, I found this product on this influencer’s Instagram,” or, “Oh, I read about this product on this particular blog.” That’s when it clicked that, “Wow, these mediums don’t work in silo. They all work together. You need to have multiple touch points in order to really convince someone that your brand is legitimate and that they should be buying from you.”

Felix: All these channels, it makes sense that you want to touch them on all these different points, all these different mediums. Of course this takes a ton of time. It sounds like you devote a ton of time to this, lots of effort and of course lots of capital as well. How did you manage all of this? Do you remember how long it took or if you could share how much budget it took early on before you started seeing this traction?

Megan: Gosh. I think from the time we took back control of the paid media and some of the influencer stuff, I would say before … I remember sitting down exactly 30 days after I started and showing the founders, “Okay, here’s what’s happening. Here’s what happened the first week and here’s what’s happening in week four and see how while we’re not necessarily crushing it, we are seeing a really upward trend of our return on ad spend and we’re really seeing this ROI very slowly creep in.” They kind of looked at me like, “Yeah, yeah. Okay. Whatever.” Then basically I would say two weeks after that things really started to take off. They realized, “Wow, okay. All this stuff is working in conjunction and working together.”

It took awhile. It took a lot of budget and it took about six weeks. What my recommendation would be for anybody who’s testing this out is to really time it out so that all of these different efforts are hitting at the same time. That was the key. Honestly, I don’t think anyone really realized how important that was, at least I didn’t. I thought we would test the different channels, the different mediums, and say, “Okay, influencers, we tested this this month and we saw this return. Traditional PR, we tested this this month and we saw this return. Paid advertising, same thing.”

It wasn’t until we actually did it all at once that they all helped each other and things really took off. My best advice, especially on a limited budget, is to really make sure that everything’s hitting at the same time. I mean, even the same day would be amazing. I know that’s easier said than done, but just really honing in and making sure that you can see your brand in multiple touch points at the same time, as opposed to testing each channel individually.

Felix: For you, you would recommend, I guess this probably varies from product to product, industry to industry, but for you in your experience, you would rather split up a budget in half and devote it to multiple channels than doubling down on just one channel. Hit that same potential customer twice. Is that correct? From that, you’d rather split it up in two different channels than hit them twice from the same channel?

Megan: I really would. I would. Especially in beauty, just because it’s so highly competitive and there are so many products out there, I think for women who are so in tune with advertising, they’re seeing it everywhere and they’re out looking for their new products and they’re so inundated with so many different things that I think it’s really important for women that they’re seeing your brand in multiple touch points.

Felix: Makes sense. I think one thing that you mentioned earlier about how a lot of these agencies that you’re working with were giving you essentially benefits that could really be applied to any brand, any product, because things like free shipping, things like discounts. All these things can be applied to any brand, any product. You recognized that you can’t lead with that because it doesn’t talk much about the brand, about the brand and the products’ benefits directly. Now, this testing process that you went through to understand what kind of messaging to create, talk to us about this. I think this is something that a lot of entrepreneurs struggle with is to how can they speak about their brand to bring out the benefits of their product in a clear way? How is somebody going to be being to go down this process of identifying how to message their product?

Megan: I think that’s a great question. I think for most new brands, people don’t just create products out of nowhere. They create products to solve a problem. They create products because they see a void in the space. I think really highlighting how your product fills that void is just so important. I think that, again, no one’s going to buy a product just because they get free shipping on it. You have to convince them that they need the product in order for them to actually determine that they want to buy the product. Then the benefit is that they get the free shipping. That’s not a reason to buy a product, though. No one’s going to do that.

I think that really differentiating yourself from the other products on the market and also just trusting your gut, if you really believe in your brand, and that was one thing I just didn’t see with a lot of the partners we worked with early on is they didn’t buy into the brand. They thought we can sell anything and this is the blueprint for it, but there’s really not a blueprint for it. Different benefits work with different products and when we really doubled down on here’s our brand, here’s our brand messaging and here are the benefits of our product, that’s when people really resonated with that and decided to try the product.

I think definitely take all of the feedback that you’re getting from your friends and family or from your consumers or everyone, but take it with a grain of salt and don’t shift gears just because your cousin told you that she doesn’t like the color of your packaging. That’s the worst possible thing you can do, especially if you’re very passionate about the products that you’re creating.

Felix: Now talk to us about tactically, when you sit down to run a test, whether it be at the very beginning or even today, how do you think about it? How do you organize, or how do you set up a test?

Megan: That’s a great question as well. For me, we have a lot of very opinionated people at Kopari, a lot of people who are really passionate about the products and about why do they love it. The great part about our brand, but also one of the challenges that we have is that there are so many different benefits to the products. It’s multitasking, which is really great for some people. It works very, very well and better than most other products, which is great for other people. Some people just love the fact that it’s a very natural product and they’re not putting any chemicals in their body. Because of all this, what do lead with?

That’s been a big challenge for us over the past 15 months or so is to figure out why are people actually buying this product. Of course, it’s probably a combination of the three. What we did was when sitting down and really just spit balling and talking about the hierarchy of our brand messaging, that’s when I took things back and said, “Why are we debating this? We could debate this at length. Let’s just test it.” We definitely tested it on our website. We also tested it in our marketing, in our advertising, just even swapping out the photos.

Do we use a photo of the product sitting on a beach to really get people to feel like they’re sitting on that beach when they use this product and smelling the coconut and everything, or is it better to use the product, or see the product when it’s actually being applied onto someone’s skin, or is it better to look at it when it’s actually in a studio that can relate to more people and during the winter? What is the best creative and how can we really get that out there? That’s been probably the most consistent test that we’ve run along this entire process, both on the site, on the digital advertising, and we’re learning something new from that every day. Even just how old should the model that we use be?

Felix: Now when you are running these tests offsite, is it mostly done through Facebook ads?

Megan: A lot of it is done through Facebook ads. We also run advertising on Pinterest, on Google AdWords, definitely a lot of display advertising as well.

Felix: Cool. When you are running these tests, you make sure to only … Tell us a little about this. Do you change just one element at a time, like just change the copy, just change the image, or do you try to make multiple tweaks at one time?

Megan: Definitely a mixture of both. If we have a really strong theory that we want to test about we believe that people are buying our coconut melt because it’s a multi-tasker, or it’s just because it’s all natural with no parabens or sulfates or anything, then we’ll take the same imagery or video and we’ll actually just change out the copy and really just test, “Okay, what is actually resonating here?” Whereas sometimes, as someone who’s been testing for a long time it’s probably bad to say, but a lot of times you get quicker results if you really just change the entire thing up and change up the photo, even just change the product that you’re showing and vastly different creative and then work backwards into, “Okay, so that definitely performed, but why did it?” Work backwards by then switching out the copy and the imagery to narrow down what it was that actually resonated.

Felix: That is a great point, to work backwards, to just blow it out the water and start with something brand new and then pull away the layers to determine what actually makes a difference. I think doing that is also a little bit more, somehow it could be more motivating too, because sometimes it’s a slog to just change one thing at a time, every single day, make one small tweak. Now when you do have a test ad out in the market, how do you determine if it’s a winner or not? How long do you wait and what are you looking at?

Megan: If you’re using a tool, I mean there are tools out there such as Optimizely or Dynamic Yield or the different tools for the onsite testing that they’re going to tell you. They’re running all the statistical analysis on the backend and they’re going to say, “This is a 95% chance of actually being statistically better performing than its counterpart.” That’s a very, very, very helpful tool because it’s doing all the heavy lifting for you and you don’t necessarily have to put a lot of thought into what’s going on, whereas when you’re doing it with advertising, there are tools that will help you and manage these tests for you but we’re not currently using one that’s actually going to say, “This is a clear winner. Send all your traffic here.”

It’s really just a matter of not only allowing enough time to go by. Usually we try not to let it go out for more than just a couple of weeks. The thing with advertising is you get ad fatigue. You can’t really just find one winner and let it run for six months. That’s not going to work. You’re actually going to have to continue to just … That’s why the testing process is just never finished. You never see one ad that’s just repeated for any brand and they’re always running multiple ads and multiple audiences to really see what’s resonating at what time during what point in the funnel. It’s kind of a never-ending process, which can be a little daunting. Lucky for me I love it.

Felix: Obviously an ongoing process that once you determine a winner, it’s not just, “Okay, this one won and now let’s move onto the next test.” You actually want to make those changes in your messaging, on your site, through all of your branding as well, I guess depending on what you’re testing. How do you manage that? How do you roll out a successful, if you determine a winner for a messaging, an image, something wins, how do you roll it out to the rest of your assets?

Megan: That’s definitely a challenge, especially as our internal team has been growing, it’s a lot more difficult to actually let everyone know, “By the way, this is the winner and this is what we’ve determined works better,” and then the opposite. Also, it’s not just a matter of A works better than B. It’s also a matter of what else is going on in the market, what time of year is it, what other advertising are we doing that could potentially be affecting this? There is no real clear cut answer, but what we’ve done internally to try to combat that is to sit down, we actually sit down every other week, and we just have a conversation about it.

We say, “Over the past two weeks, here’s what’s been working. We’ve seen this perform better than that.” We’re actually working with the content creation team directly and saying, “See this, see that.” Now, as far as the digital marketing team, we’re very into numbers and analytics and then the content creation team is essentially taking this and taking our recommendations and saying, “Okay, if this worked, why don’t we try to get more video that looks like this or more photography that looks like this.” Then they’re taking our feedback and going out and getting new content to create that then we can go back and cycle through.

Felix: I like that, that you work with the content team because it needs to change. You can’t just let the content team run off without the analytics that you can provide to guide them on which direction to go. You mentioned a couple of products, Optimizely, which is what I’m familiar with for your onsite AB testing. Talk to us a little more about this. What does the software do and how do you use it?

Megan: A tool like Optimizely, or Dynamic Yield, or any testing tool that goes onto your website, I will admit a lot of them are very similar. What you do is essentially, just install a line, a java script, onto your backend. Then you use the actual dashboard within the tool to create your tests. It actually goes in and sits on top of your code. It’ll change the image or the text or the order or the design of the page directly on top of your actual page. You’re not making these changes in the backend of your website. You’re actually doing it in their WYSIWYG Editor through their dashboard. Then you’re setting up the tests, you’re setting up who’s actually going to see this test, who’s going to be included in the test, what your goals are.

Nine times out of ten our goal is always going to be which one increases revenue the most, which variation increases the revenue the most. Then it’s actually going to go in, as soon as you start your test, and auto divide your audience up randomly to determine who gets to see what variation and then essentially calculate the statistical significance of the results of the test, so which variation did generate the most revenue per visitor and how likely is it that this isn’t just due to chance and that this is actually a significant result and A is actually better than B.

Felix: Can you talk to us a little bit about a change that you’ve made recently to the site that has had a big impact on the