Revolutionizing UX and product development – Q&A with Casey Winters, former growth team product lead at Pinterest

| May 1, 2017
Sign up to get weekly resources, and receive your FREE bonus eBook.
Thank you!

Get ready for some great content coming to your inbox from the team at UserTesting!

In a recent webinar, Casey Winters, the former product lead for the growth team at Pinterest and advisor to multiple growth teams at other companies, talks about how growth teams came to be, how they operate at scale, how the user experience challenges are different, and some effective experiments on specific channels he’s seen in his career. We had a great Q&A session with Casey and included some of our favorite questions below, or you can watch the full webinar here. Enjoy!

What tool or tools did you use for A/B testing at Pinterest? Was it built in-house or something you purchased?

Pinterest in a unique situation in that very early, Pinterest created a heavily-engaged user base and there’s a lot of users. A lot of off-the-shelf solutions wouldn’t really work at Pinterest scale, even though we were a fairly young company.

We developed our own A/B development framework in-house tied to the key metrics that we cared about like sign up, re-pins. We actually built an experiment framework for SEO to track traffic increases. That was all built in house.

At GrubHub we use tools like Monetate and Optimizely and they worked pretty well. Eventually, there are some significant advantages to building your own.

What other platforms or tools are you guys usually using as a growth team?

It definitely depends on the size of the growth team. Pinterest definitely had a culture of building most things in-house because of the scale of usage.

For example, we looked at using an email provider to be able to send out emails, but Pinterest is sending multiple billion emails a month, all with personalized content for each user. Being able to send data back and forth to a third party at that scale is really difficult. We had to build something in-house. It took nine months. It was hard work, but we felt good after we had built it.

At GrubHub we used I believe it’s called Salesforce Marketing Cloud. We knew it as Exact Target. That worked really successful for personalized emails to our user base. We used some tools for SEO tracking.

There’s this took called Get Stat, which helps track your SEO rankings on a daily basis. That’s been very helpful for us. Most of the time, Pinterest is solving its own problems because the scale question makes it hard for third-party solutions to work.

What was the lift and conversion on app versus web at Pinterest?

Actually, the conversion was lower, but the activation rate of users once they did sign up into sticking around for the long term was three times higher on the mobile app versus the mobile web.

Given the decrease in conversion rate, that created a scenario where you got 1.3 mobile app users for every one mobile web user if you do the math, subtracting out the conversion rate decrease, but the activation increase.

Does the forced app download on mobile impact your mobile web legitimacy? Would it lead to an increase in frustration among users?

These are really tough questions that you have to have at the business level. The reality was the mobile website was already suffering. That’s why the metrics showed the pushing people to the app was a much better experience because it had a lot more investment.

If you’re in a perfect world, you want to have an amazing mobile website, you want to have an amazing desktop experience, you want to have an amazing app experience.

You always have to make tradeoffs on how many platforms you can support. Given that mobile web wasn’t getting a lot of support and the app was working really well for the user base, the decision was, “Let’s close down mobile web more so we can get people to the right experience.”

It does create that short-term frustration, but that long term more valuable user experience. We really always tried to optimize Pinterest for what creates the best long-term experience for users, even if there’s a little short term pain. I think of it like it’s a little bit of a tax you have to pay to get the true value of something.

You have to figure out what friction you’re comfortable introducing to the user and make sure that you get the long-term benefit from that. If you’re introducing friction without creating benefit, that’s when you’re in a really bad position and you need to really think why you’re doing that.

How long does it take for an idea to actually make it onto the website or app?

For each one of those individual iterations that I showed, Pinterest probably ran the experiment for about a week to get statistically significant data on what was going on. Now, if you’re at a company that gets less traffic than Pinterest, you might need to run it for a month or longer in some cases.

Generally, we’d run it for about a week or two, regroup, do an experiment interview, make sure we understood what was going on, why the metrics would look the way they are, and make a decision about, “Do we iterate on this? Do we kill it and move on? Or do we want to ship what we have?”As you saw, we did probably 10 or more app interstitials. You’re looking at across the course of six months to a year all of the different work that we tried.

How do growth teams discover where to start first? How do you identify areas of your product that need help?

What I advise people thinking about starting growth teams and side companies is to focus on one problem at the beginning. There are a few problems that are usually the best places to start. The ones that I tend to look at are ones that are frequently ignored by core product teams, given how product teams are incentivized.

One is the logged out experience. How do non-users experience the product? That is one area that typically is given a little bit less love, but also has a lot of people visiting it so you can do experiments and learn quickly.

Another one is emails and notifications. You have the entire user base at your disposal and it’s outside the product, so it’s generally something that the core product team thinks about less.

The third is referrals. These are usually pretty easy to try a few things after people are already getting value. Then also those quick iteration cycles on, “I did this thing, I saw exactly what happened, and I learned, and I can move on.”

In terms of how to find that problem, you have to dig into your data and see what the issues are. When I started at Pinterest I was actually there to work on SEO, but I noticed this conversion rate was lower than one percent on these pages. It’s like, “Why are we focused so much on driving more traffic to these pages if these people aren’t really getting introduced to the real value of Pinterest through that?”

By doing that analysis and seeing that a lot of traffic low conversion rate, if you improve that a little bit that has a huge impact on a company, I was able to find that that’s one of the first areas I should look at.

What roles and what skills do people have that typically make up the entire growth team?

Typical growth team includes a cross-functional team of design, product management, engineering, and analytics. If you’re starting to do paid acquisitions, sometimes marketers will be involved in that process as well.

What we’ve seen as the DNA of a successful growth team member is that they’re more focused on impact and outcome, instead of technical complexity, especially in engineering. They’re people that are more focused on the results than their specific idea. If someone is really wrapped up in their idea and gets tied to it, that’s not usually a successful way to run a growth team because most ideas are going to fail, or they’re going to need to morph significantly. You have to just be willing to find the right one.

Then the other is comfortable with pushing things out that aren’t pixel perfect. Everyone wants to create the most amazing experience where every single thing looks perfect, but if you do that and the product direction is bad, then you’ve wasted three months. What the growth team is really focused on is, “How do I do the minimal amount of work so that I can push something out to users to validate an idea and then get comfortable iterating on it and polishing it later once I’ve verified that it’s actually the right thing to be working on?” Some people just aren’t comfortable with that.

As a result, a lot of growth teams start with more junior people because they haven’t had a decade of working on some of the other ways of thinking, and engineering, or design, which is more hard technical problems or pixel perfect artistry. They can learn from scratch the growth way instead of trying to unlearn some traditional ways that don’t work as well for growth.

How did you get buy-in from stakeholders that’d be willing to push out non-pixel perfect designs live to your users? Did you encounter any resistance? If so, how did you overcome it?

Pinterest is a very design-focused company. One of the founders is a really brilliant designer. This was not something that Pinterest was really comfortable with at the beginning. When I came in I was asked by the executive to prepare a product strategy for the growth team. What I actually delivered was an operational strategy. That operational strategy talked about why growth needed to be iterative and how we would escape local maxima thinking, which is a common issue with growth teams getting stuck in design that they can’t iterate out of.

I basically went in on a very long filibuster to our CEO, and our co-founders, and our execs around why the iteration matters and creating a process where we can iterate on a percentage of the user base with something not polished, but by the time we want to get it to 100 percent of the user base it will be polished.

For example, in those experiments, we would ramp from showing it to one percent of users to 25 percent of users. At that point in time, we could usually get enough statistical significance to say, “What was happening here? Was this an experiment that was working or not?”

Then we would do the experiment review to decide whether we were interested in shipping it or not. At that point, we would go to our creative director and say, “Here’s the experience, here’s what happened in the experiment. We would like to get this type of experiment out. What kind of tweaks and polish do we need to make so this fits Pinterest design standards?”

We would make sure we would always do that step before it actually just released to everyone. Usually, we found that that was just a couple subtle tweaks. It wasn’t like we needed to do an amazing amount of work, but there was definitely work there.

What do you use for getting qualitative app feedback? Are there are any specific learnings that are especially important that you can get from qualitative versus more quantitative data around user behavior?

We use UserTesting, of course. We actually use that at GrubHub as well. Basically, with qualitative research, we’re either bringing people into our office and watching them use the product or we’re using UserTesting.

In terms of what data we get from qualitative research that we don’t get from the quantitative experiments that we run, quantitative experiments help you understand what people are doing, but they don’t really tell you why. If you want to understand why an experiment is or is not working, you have to watch users, and you have to have them explain themselves, and you have to ask questions.

I find that most growth teams start without that function, but especially if you’re going to start doing a lot of design related work, designers really need it to create the deep insight on what’s working for growth. The engineers need it as well they just don’t realize it. It’s been a super critical part that we’ve added over the last year and a half to the growth team at Pinterest and it’s been super helpful for us in finding especially new opportunities we hadn’t considered. The feedback you get from real users is just hard to get in a dashboard.

What does your team’s brainstorm process look like when you’re attempting to optimize a problematic feature?

In general, we do an OKR process at the high level. Where we think about, “What are some major themes that we want to work on, some major problems?” Then we brainstorm with the entire team. We brainstorm with engineers, designers, product managers, analysts in that specific area of ideas. Then we’ll prioritize ideas based on of the level of effort, the level of impact we expect, and then the chance of success. That will get a board list.

Once we start going into, “Hey, here’s an idea we talked about to solve this problem,” we’ll do an experiment kickoff with the designer and engineer that are going to be focused on that project, and then generally the leads the team, which is a design and PM-lead.

Then it’s up to the designer and the engineer working on that specific project to go off and brainstorm solutions and bring them back to the broader team. They can include other people in that brainstorm if they want and a lot of times they do. They’re expected to work out solutions that they will bring back and propose for the problem.

The reason we designed it that was is that a lot of times if designers and engineers aren’t working together on this, one person will try to lead too much and it will be something that’s beautiful, but way too technically complex for the experiment. Or something that’s very bare bones designed or not a good user experience.

You really need to have them build that partnership at the project level, we found, to be long term successful.

Should we think differently about growth teams in large versus small companies?

In small companies, you’re typically focused on just trying to prove that a growth team can work on one problem set. You have maybe one designer, one engineer, maybe a PM, maybe an analyst working on something.

At large companies, you have a lot of these different sub-teams. You might have multiple engineers and designers on different teams. Then especially if a growth team is very mature, you’re working in deep areas of the product. For example, at Pinterest, getting more people to re-pin is something the growth team could work on even though that’s a core feature, so completely redoing the flow.

The other thing that’s interesting about large companies is that you might have competing products. Small companies tend to have one product and everyone is focused around making that product better and growing the use of that product. At very large companies you might have a dozen different products in your portfolio and not only do you need to think about growing all of them, you need to think about which one you might prefer growing over another.

You want to make sure you have a clear understanding of what takes priority in that case and it needs to be based on math. Every product manager is going to think their product is more important than other products. You need to have a way to negotiate those situations based off of data. That’s a key thing that early growth teams don’t need to think about, but once you have multiple products can be a real problem in understanding.