How to Approach Experimentation: Bite-sized Changes vs. Big Swings

Growth by Seeking Wisdom podcast

Today on #Growth, Matt is talking all about experimentation. Specifically how to approach changes you might make to your site or product – bite-sized or incremental changes vs. going all out with big swings. Matt breaks this down with a real-world example you’ve probably faced yourself – updating your homepage. There are pros and cons to both and Matt lays it all out today on #Growth.

You can get #Growth on Apple PodcastsSoundCloudSpotifyStitcherGoogle Play Music or wherever you get your podcasts. Or listen to the full audio version below 👇

Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️⭐️ review and share the pod with your friends! You can connect with Matt Bilotti on Twitter @MattBilotti.

Subscribe & Tune In

Apple Podcasts Spotify SoundCloud

Full Transcript

Matt Bilotti: Hello, and welcome to another episode of #Growth. I am your host, Matt Bilotti, the guy with the beard, although by the time this episode airs, the beard will probably be gone because I mostly grow it out as a winter coat type thing to keep me warm on my walks to the office from home, but as soon as it starts getting warmer, I buzz the whole thing off. Anyway, I digress.

Today, I want to talk about two different ways to approach changes or experimentation. The two different ways are bite-sized changes or big swings. Let’s take this in the frame of an example. Let’s say you want to improve the conversion rate of your homepage, and so you can make a bite-sized change, and you could the CTA on the page to say, “Get started,” whereas it used to say, “Sign up for free.” You could make that change as an incremental change, and at the end of the experiment, you’ll really easily be able to point back to it and say, “This number moved because we changed this copy.” You can isolate perfectly what the thing was that changed.

Well, your other option is, is to do what I call big swing. This is much more of an overhaul. Instead of changing the CTA on a homepage, you change basically everything. You change all the variables. You design a completely new page. You have a different headline, different CTAs, different colors, all that stuff. At the end of it, it gets a little bit harder to isolate exactly what it was that made the difference. Let’s say you ran a test, and it was 5% better. It’s hard to say was it the combination of all those things, was it the headline that people connected with; however, you’re far more likely to see a bigger change than when you make an incremental change. An incremental change or a bite-sized change is going to give you an incremental improvement, a big swing, or a complete overhaul is way more likely to net you a better result. I want to talk a little bit about one example that I have from here, Drift, and then I also want to talk about how you choose which one to do and when to think about that.

We have a chat widget, if you ever signed up for Drift or saw it, you probably have because you’re listening to Seeking Wisdom, and if the chat widget’s installed on a website, it has at that bottom a link that says “Built by Drift” or “Powered by Drift,” and clicking on that would take you back to a landing page on Drift’s website to recommend that you sign up or chat with us or check it out.

We had those pages going to, that was the URL, and so we looked at it and said, “This is actually one of our higher-traffic pages, and no one’s done anything with it since we first built it.” We first built the landing page two and a half years ago. We threw it up there. We grabbed some copy that we had from the homepage and made little bit more of a personalized type experience coming from a chat widget, and we put it there, and it was great.

We’re looking at this and saying, “Wow, conversion rate is in the low single digits. Is there any way that we can get this higher?” We have two options. We can make bite-sized changes and change headline, change copy, that kind of thing, or we could take big swings. We said, “You know what? We’re at a starting point, so let’s take a big swing.” A big swing is going to give us a new starting point, rather than saying that, “Oh, we probably had some good stuff to begin with,” or maybe we were looking at it and saying that the benchmark of this conversion rate is much lower than what we expect. Maybe that’s something that you’re doing when you’re looking at this. Then it’s time to take a big swing because you’ll get a bigger result.

What we did on those pages, I’ll tell you exactly what we did, we had a version where we removed as much friction as possible. This used to be just a regular landing page, but we said, “What do we want people to do on this page?” We want them to sign up for the product because they just experience the product. What’s the best way to do that? Well, they show up to this page right now, and it’s pretty easy to get distracted, so what if we remove the header, what if we remove the footer, what if we made it so you can’t scroll on the page, and what if all that there wants to do was to enter your email to sign up and copy that said, or headline that said, “Want a chat widget like you just saw on” That’s not the exact copy that we use, but it’s pretty similar. We roll that out as a big swing, saying, “If it doesn’t work, whatever. We’ll try something else and build off the other version that we had before.”

However, we actually wound up getting three to four times in increase in the conversion rate, and that’s because we took those big swings. That, to us, was a moment where we said, “Huh, now we have a new starting point, and the conversion rate’s just jumped a ton. We probably don’t need to take another big swing because this thing exceeded all of our expectations, and now, let’s try some bite-sized changes.” Then we took this new version and we tried changing the headline and we tried changing the CTA and we tried changing the colors and the social proof. That was on the right side of the page. Then we started doubling down into the bite-sized stuff.

When you’re thinking about should I bundle a bunch of changes together or should I make one single change to isolate the variable, the important thing to think about is how far below where were want this number to be are we right now. If you’re trying to get in conversion rate to 10%, and right now, it’s at 2.5%, you should probably take a bunch of big swings. You’re not going to be able to just make bite-sized changes over and over. I mean, you can. It’ll just take you a long time to make those changes over and over to go from 2.5% to 3% to 3.7% to 4.6%. There’s going to be a lot of trial and error along the way.

When you’re thinking about what should I bundle here, look at the difference between where you are now to where you want to be, and if that is a two or three-X type change or a very significant raise in a conversion rate, if you’re at a 70% conversion rate, and you’re trying to get it to 77% conversion rate, you should probably make some bigger changes and make big swings because that’s where you’re going to find all the value of a new variable.

Now, of course, you’re going to exit the experiment and say, “I don’t know, we changed a bunch of stuff. What was it that worked really well?” and that’s when you can then back into, “All right, let’s now swap out the headline and see if that is it,” but at the end of the day, most of the time, it kind of won’t matter because you have a new normal, you have a new baseline, and then you get to build off of that. If you’re trying to get incremental improvements, then make bite-sized changes. Those will get you there without risking losing a huge drop off rate, but if you’re way below where you want something to be, then you just gotta keep taking those big swings until something works, and realistically, try seven different things, and maybe one of them will work.

The example that I just talk through on our powered-by pages with the no header, no footer, all that, that was one big swing in a bucket of five different big swings that we tried all at once. It just happened to be the one that worked really well, but it wound up being three times better because it was so different and such a big swing that it wound up panning out.

That’s what I got for today. Thank you so much for listening. If you have ideas, feedback, anything at all, my email is I would absolutely love to hear from you and hear what I should be talking about next. I will catch you on the next episode. Thanks so much. Bye.

Related Stories

P.S. Join 20,000 of your peers. Subscribe to the newsletter for hypergrowth.

Every Sunday evening we'll send you a roundup of the best content and events from Drift and around the web. Make sure you're ready for the week! Subscribe now.

Subscribe Here