If this is your first time reading, I recommend you start with my 6-month challenge and table of contents of weekly posts for the past 11 weeks.
tl;dr This week I met with four beta testers to do user testing for Cusoy, much to positive feedback but also repeated comments on certain things to iterate (which is good!). I plan to make those iterations, then meet with another additional five beta testers (all different ones) for follow-up user testing and then prepare for a soft launch.
So this post will be split mainly in two parts —
- First, I’ll talk about how many users you should test with, user testing and iterative design and how to build great products by prioritizing your features into different buckets; and
- Second, I’ll discuss more about other personal things that aren’t so much related to Cusoy’s product development process, including a video of my interview with Jason Shah last week on my background on breaking into product management as well as a recent video on rejection that really resonated with me and how rejection has shaped my life trajectory so far.
If you’re just interested in Cusoy’s progress, the first three sections will be all you need to read.
If you’d interested in learning more about me as a person and my background, the last two sections will provide extensive color on me. Feel free to skip the last two sections if you’d like, too. It’s a bit long, but I enjoyed writing it as a somewhat cathartic exercise.
I. How many users should I test with?
II. User testing and iterative design
III. How to build great products
IV. On breaking into product management – as a non-technical undergrad to Evernote APM
V. A word on rejection
VI. Next week and key lessons learned
9. Engage Users
Product development is a conversation with the user that doesn’t really start till you launch. Before you launch, you’re like a police artist before he’s shown the first version of his sketch to the witness.
It’s so important to launch fast that it may be better to think of your initial version not as a product, but as a trick for getting users to start talking to you.
I learned to think about the initial stages of a startup as a giant experiment. All products should be considered experiments, and those that have a market show promising results extremely quickly.
Once you start talking to users, I guarantee you’ll be surprised by what they tell you.
When you let customers tell you what they’re after, they will often reveal amazing details about what they find valuable as well what they’re willing to pay for.
The surprise is generally positive as well as negative. They won’t like what you’ve built, but there will be other things they would like that would be trivially easy to implement. It’s not till you start the conversation by launching the wrong thing that they can express (or perhaps even realize) what they’re looking for.
– Paul Graham, What Startups Are Really Like
I. HOW MANY USERS SHOULD I TEST WITH?
I know I mentioned in a previous post I was planning to do 15-20 user tests over the next 1-2 weeks, and even the EIR at Rock Health recommended I meet with 20-30 users — but I’ve changed my approach a bit.
Figuring out how many users to test
I’ve decided to use a small sample of 5 users per each “round” to test. Not to mention, trying to schedule 15-20 people in two weeks is a lot harder than 10-12, and these additional users most likely will be telling you things you’ve already heard before in previous user tests, speaking from direct experience!
A famous Nielsen article (2000) explains why you need to test with 5 users:
Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.
As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems.
Of course, this is an extremely general principle — and Nielsen includes a caveat that “the formula only holds for comparable users who will be using the site/product in fairly similar ways.”
Others have also chipped in on the somewhat controversial magic “5” number:
The five user number comes from the number of users you would need to detect approximately 85% of the problems in an interface, given that the probability a user would encounter a problem is about 31%. Most people either leave off the last part or are not sure what it means. This does not apply to all testing situations such as comparing two products or when trying to get a precise measure of task times or completion rates but to discovering problems with an interface.
– Jeff Sauro, Why You Only Need To Test With Five Users (Explained)
Laura Faulkner conducted a good in depth study of the 5 user rule and shared the results in her paper “Beyond the 5 user assumption: benefits of increased sample sizes in usability testing.”She proved that confidence intervals in samples of five users were around 18.6%…ouch! She concludes:
Although practitioners like simple directive answers such as the 5-user assumption, the only clear answer to valid usability testing is that the test users must be representative of the target population. The important and often complex issue, then, becomes defining the target population. There are strategies that a practitioner can employ to attain a higher accuracy rate in usability testing. One would be to focus testing on users with goals and abilities representative of the expected user population. When fielding a product to a general population, one should run as many users of varying experience levels and abilities as possible. Designing for a diverse user population and testing usability are complex tasks. It is advisable to run the maximum number of participants that schedules, budgets, and availability allow. The mathematical benefits of adding test users should be cited. More test users means greater confidence that the problems that need to be fixed will be found.
– Marie-Claire Dean, Usability testing: 5 users is not enough
The benefit you get from adding a few more users to the total (or in the case of 5 users, doubling the amount) is far greater than the small test that gives you “quick and dirty” results. In the case of running a series of usability tests or iterating your testing process (recommended for refinements based on evolving design decisions), you may want to choose a smaller number of users: I recommend no less than 8 users.
– Frank Spillers, How many users should you test with in usability testing? (Latest Research)
Nielsen then wrote an updated article in 2012 further elaborating on his previous 2000 piece. He notes several exceptions here but still asserts that 5 users are enough:
The answer is 5, except when it’s not. Most arguments for using more test participants are wrong, but some tests should be bigger and some smaller.
If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you’d find using many more test participants.
This answer has been the same since I started promoting “discount usability engineering” in 1989. Doesn’t matter whether you test websites, intranets, PC applications, or mobile apps. With 5 users, you almost always get close to user testing’s maximum benefit-cost ratio.
– Jakob Nielsen, How Many Test Users in a Usability Study?
II. USER TESTING AND ITERATIVE DESIGN
User testing approach in iterative design
For my purposes, my approach is to ship MVP v3, test with 5 users, iterate on their feedback, ship MVP v3.5, test again with 5 users, iterate on their feedback, and then do a soft launch — and potentially do one more user testing round with another 5 users.
Note: these are all users who are gluten-free by necessity (gluten intolerance or celiac disease), not users who are gluten-free as a healthy lifestyle choice. It would be good to test with the second segment of users soon as well.
First “round” of user testing
For this first “round,” I was specifically testing for 1) overall general usability on desktop and mobile web and 2) several specific things: understanding of colored clusters on the map view, search/filters order in top search box as well as the order and relevance of information on a listing page.
These were in-person meetings held in Berkeley, San Francisco, Millbrae and Foster City.
While I brought my Macbook Air to show them the desktop version and test the desktop app that way, for mobile web each of them pulled out their smartphones (iPhone or Android) and interacted with the app in their natural environment with their own smartphone (as opposed to testing on my “slow” 3G iPhone 4). I thought this was an important point to make — if possible, you always want to see the user interact with your product in his or her natural environment.
Overall feedback
Everyone pretty much provided positive feedback. It’s a clean, usable and user-friendly design that they were excited to see once it goes live, and even one user, while interacting with the search filter on mobile web, point blank said, “Yeah, I’d definitely use this.” 🙂
There were some repeated feedback:
- Obviously, add more restaurants till you get to 50 (interestingly, this was the minimum number mentioned multiple times by users)
- Add icons/symbols at the top of the listing page to indicate gluten-free kitchen, gluten-free certified, vegan, gluten-free menu, etc.
- Add gluten-free notes at the top of the listing page rather than menu items first
- Have the high-level highlights of each listing at the top of each listing page — most important things a user should know first, even in bullet point form
- Add user ratings/reviews (preferably with specific categories — such as cross-contamination practices, staff awareness, etc)
- Add form for users to submit listings
- On mobile web version, let the user first see what he or she is tapping on in map view instead of immediately directing a user to a listing
These are all fixable things I can do before I conduct my second round of user testing. In addition, they also gave me some helpful tips on distribution strategies once I launched.
Second “round” of user testing
For my second “round” next week, I will not only be testing the above (to a lesser extent), but also be testing user submitted reviews/ratings as well as user submitted restaurant listings. I am also interested in testing completion of actual tasks rather than a general exploration of the app’s usability.
Some difficulties
One of the difficulties I found when recruiting users was not their enthusiasm, but that since I made it clear in an email to them that Cusoy will only be serving San Francisco (for now), that may have caused several users to not be open to meeting because they live in East Bay, Peninsula or South Bay where they might think meeting may be probably pointless.
This is false — as even if Cusoy only supports SF listings, it is still testable and usable for users who may not live in SF or eat in SF. Ah well. It’s not a big deal, but something I probably could have done a better job framing to help convince users to help user test it even if they don’t live in SF.
III. HOW TO BUILD GREAT PRODUCTS
So I came across this article on how to build great products by Slava Akhmechet which I enjoyed so much that I wanted to include my key takeaways here.
Akhmechet advocates a model of categorizing features into three distinct buckets:
A gamechanger. People will want to buy your product because of this feature.
A showstopper. People won’t buy your product if you’re missing this feature, but adding it won’t generate demand.
A distraction. This feature will make no measurable impact on adoption.Empirically, successful products have one to three gamechanging features, dozens of features that neutralize showstoppers, and very few features that are distractions. Your job is to build an intuition about your space to be able to tell these categories apart. That’s still pretty subtle (is a built-in phone projector a gamechanger or a distraction?), but at least this model gives you a plan of attack.
Which brings me to think… what are gamechanger features that I can offer through Cusoy? Is it the information piece? Is it the ratings/reviews? “Users” here can apply both to either users or restaurants as customers. Hm, I’m not sure, but I’m on the way to finding out, slowly but surely. I think. I hope.
I’ve made it an absolute priority to not having any showstopper gaps — it’s been abundantly clear to me that it seems the use case for Cusoy will be 75-80% mobile, with the remaining 20-25% desktop use (if any). So the fact that Cusoy is mobile friendly and responsive is extremely helpful, even if it’s not a native iPhone or Android app (yet).
Overall, Cusoy is extremely usable, so the showstopper part is not as applicable to me (or at least, not at this MVP level).
In terms of “distraction” features, I’ve been trying to do my best on not spending time on any distraction features. The only things I can think of are integrating beta invite codes when a user signs up (to keep out users who won’t derive direct benefit from Cusoy from registering), but decided to only have users register if they want to post a listing and/or rating/review.
But generally speaking, every feature in Cusoy is there for an explicit, essential reason and I don’t waste my time adding additional features if it’s not going to make a measurable impact on adoption.
IV. ON BREAKING INTO PRODUCT MANAGEMENT – AS A NON-TECHNICAL UNDERGRAD TO EVERNOTE APM
Last week I hopped on a Google Hangout with my friend Jason Shah, a PM at Yammer, to contribute a brief interview for his Udemy course on how to get a job in product management. He also interviewed other PM’s at Asana, Pinterest, Yammer, Dropbox and others.
I had started blogging about how I got my dream job (part 1) awhile ago — part 1 was finished but I never got around to finishing part 2, because I wanted to talk about the art of networking, go through the 75-100 conversations/notes I’ve had with people I’ve met along the way and distill all of that into actionable advice. Part 2 probably won’t be written until next year.
This video talks a little bit about part 2, but only scratches the surface.
Here’s the video of my interview if you’re interested in watching:
Some notes about this interview that I forgot to mention and feel terrible about:
I actually was super nervous in the beginning. Maybe it didn’t come out, but I felt like I was stumbling over my words and didn’t provide enough details as I wanted — for instance, when doing sales/marketing at Square, I realized I wanted to impact the product itself and sales was one step removed from the product. That was how I first learned about product management.
Or also, talking with different users when evangelizing Square’s product, I got to learn about different use cases and approached very interesting questions (to me at the time) of, like, if Square serves so many verticals (flower shops, restaurants, salon owners, bakery owners, student clubs, coffee shops, etc) — how do they prioritize all the different feature requests and nuances coming from all these different use cases? Square could choose to dive into the restaurant space, but if Square caters to restaurants and builds features specifically just for restaurants, it obviously will lose its generalist competitive edge (to acquiring as many customers as possible) — and same goes for any vertical. So then the next thing is, how do they come up with features to satisfy the majority of their users, right? This got me thinking a ton about product.
Or talking about why I didn’t work for Square after RetailMeNot — it had ballooned up to 600 people or so then and I hated working for big companies. “Big” not like IBM big, but startup “big.” Plus, it’s a technical company (product management wise, doing software and hardware) and after speaking with product people there, I frankly didn’t think it was a good fit for what they needed and what I had to offer them. If I wanted to work for Square, it’d probably be in sales/marketing and I definitely did NOT want to do that after I graduated. So… I peaced out.
Or my comments or implications that RetailMeNot and Evernote are not “technical” companies. Sorry, I think that could’ve come out the wrong way. They are technical in their own respects, obviously, my key point I was trying to make was, in relative comparison to Google or Palantir, for instance, Google and Palantir are extremely driven by engineering. Engineers rule absolutely everything in product, sales, marketing, etc. at those places. That is why they refuse to consider anyone who doesn’t have a CS or engineering degree — you have to be technical to even begin to understand the product. Makes sense. Whereas at RetailMeNot and Evernote, they are not driven by engineering to the same degree. Evernote, for example, is driven by product and design, of delivering an amazing user experience. So again, really researching companies and being selective, understanding their strengths, what drives product management in each company, etc. and strategically weaving your story and your strengths and work experience to match their needs in a genuine way.
Or that Jason’s comment of it taking me 18 months full circle for me to break into product management, it actually took me 12 months, or a year. Again, I might not have showed it, but I was really nervous and didn’t think very clearly about what he said at that moment.
Or my comment on how I got almost all my internships through hustling — I was talking about my past finance internships, not just internships/job at Evernote (which I’ll get into later on the section of rejection), because, like I said, I have a strong trend of hustling in my life. Watching the video again, it might’ve sounded strange I said internships, plural, when it came off as only RetailMeNot as the internship and Evernote as the job.
Or that when looking for a PM internship or companies to work for after RetailMeNot, I ONLY targeted e-commerce companies… and companies whose products I absolutely loved and could envision myself working for, even if they were not e-commerce, which I forgot to mention. It was an extremely short list, by the way. Though I must’ve talked to over 50 people in many different kinds of startups, I was only seriously considering a very select few, 3-5 companies, to actually work for. Evernote was the only non-ecommerce company I looked at seriously. The more targeted you are, the higher your chances of success. Be strategic.
Or that I only like to work for consumer products — I have no affinity and no empathy for B2B products and enterprise companies. My philosophy: if I can’t personally use your company’s product(s), I don’t want to work for you because it just won’t be a good fit for either of us. Consumer products are my strengths, and not only was Evernote an app I considered absolutely essential to my life, but also they had an extremely strong focus on product and design, a huge priority on delivering an absolutely beautiful user interface and user experience.
I spoke very slowly and deliberately, because I don’t like to improvise my speaking and I was trying to put my thoughts together coherently — and articulate them in a way that made the most sense to whoever was watching. I had rehearsed my thoughts the night before and in these slow pauses, was trying really hard to remember everything that I said when I did a mock interview with myself before doing it with Jason. Yeah, I’m kind of weird like that, huh?
So all these notes post-interview I wrote above… that was what I was trying to remember when speaking slowly and deliberately with Jason. Apparently I couldn’t remember everything I wanted to say, but I feel better posting about it here now! 🙂
V. A WORD ON REJECTION
I saw this video on rejection earlier from Jia Jiang:
Jia Jiang from Chris Guillebeau on Vimeo.
This video on rejection really touched me. I laughed at times and was hanging on the speaker’s every word. Although it was clear that English was not his first language, he delivered a truly amazing and inspirational speech — but more importantly, told a terrific story. (I’m big into stories, storytelling and their impact on people).
I finished watching it with such a positive, heartwarming feeling in my heart that I haven’t felt in such a long time. I felt carefree and happy. It was a very strange feeling. It was as if a huge, unspoken weight had been lifted off my chest and I finally had room to breathe. I unconsciously exhaled a sigh of relief, because this guy articulated my own fears of rejection better than I ever could.
I thought he was incredibly brave for putting himself out there and exposing his fragility and vulnerability. I admired him and the level of respect I had for him shot up and up as I kept watching the video.
At the end of the video, he asked his audience to share with him their own stories of rejection.
…so I thought I’d share some of the tons of rejections I’ve received so far in life.
Be kind, for everyone you meet is fighting a hard battle.
– Plato
His story and journey of 100 days of rejection therapy resonated with me so much because my whole life I’ve gone through rejection. I have an extremely deep-seated, visceral fear of rejection, which at times leads me to reject myself, get depressed and suffer from low self-esteem, and yet ironically put myself in situations where I’m simply asking to get rejected… because my desire to reach my goals and “see what happens” is greater than my fear of rejection.
The mentality that “I’m scared shitless if I do x, y, z. What will people think? What will they say about me?… but life is so short, so who cares” sort of thing. It’s a sort of masochistic exercise, almost.
You might be surprised, given my past background and work experiences, but it’s true.
I myself have gone through hundreds, if not thousands, of rejections. I still keep all my rejection emails in my Gmail account — not that I am depressed over them, but just marveling at how many “No’s” I had to encounter, and thankfully so, in order to get to where I am today.
I don’t think you can grow as a person if you haven’t gone through rejection, not just once, but multiple times. In different stages and situations of your life, with a lot or a little at stake, and so forth. It is really a test of character, depending on the severity of the situation and scale of rejection.
I haven’t gotten as many in-person rejections as this guy actively tried to do during his 100 day rejection therapy (although to be fair, these were extremely inconsequential, none of the rejections were “serious” as to getting a rejected for a job, for instance) , but I’ve gotten rejections many times from people — either via email or on the phone — not interested in my services, not open to helping me, or rejecting me as an intern or job candidate.
This not only applies to me trying to recruit users to beta test Cusoy, but when I was cold-calling people trying to sell them mortgages at low interest rates (hello, Merrill Lynch private wealth management internship — back in college), cold-calling and cold-emailing at least 40-50 different boutique investment banks in Dallas/Austin/Houston sophomore year (until I got an offer from a Dallas private equity firm headed by the former chairman of TXU, involved in the largest LBO in history in 2007 — very serendipitous!), cold-emailing an investment bank not looking to recruit summer interns (they weren’t looking, but I asked anyways and ultimately ended up getting a summer analyst position and bypassing my entire competition at Rice — as they didn’t even come to campus to recruit anyone else), cold emailing all the PM’s at RetailMeNot before I got one to meet with me for a coffee chat (which ultimately turned into an interview then converted to an offer), cold-emailing 75-100 people in San Francisco before setting up 14-16 meetings back to back in 3-4 days during a networking trip in March 2012 and repeating that process and getting similar results when I later went back to San Francisco and NYC in October and November 2012, respectively.
Yes, I’ve gotten amazing and serendipitous results above, as you see. But only because I mentioned the end result, and not the process. I probably went through 50 to 75 “no’s” for every “yes” I got. Did you know that? Probably not. Is it something I like to rub in people’s faces and trumpet my failures and rejections? Absolutely not.
During this process, I must have collected at least 500+ emails and interactions over the past couple years that either went nowhere and unanswered (implicit rejection) or came back to me with a curt and polite “No thanks” (explicit rejection). It was really hard to not take things personally, I’ll be honest. These include both rejections when I was still under the impression that I enjoyed investment banking/corporate finance and envisioned a career in finance for myself, as well as in tech when I was reaching out to people in business development and product management.
Rejection is actually good for you. I’ll give you an example — I was actually -this- close to landing a cushy job as a financial analyst for BP (yes, that beloved oil and gas company) — before I surprisingly bombed my final round interview and was sent home empty-handed the next day (and actually lead me to look more into tech, my true passion, and not try to make finance “work” for me as a career).
That one really hurt — as in spite of the bad press and BP oil spill, they have truly amazing people and culture — but looking back, I’m extremely glad that happened. Because none of this — product management at RetailMeNot and Evernote, moving to San Francisco Bay Area (my dream), starting Cusoy and meeting amazing people all throughout this time — would have happened otherwise.
I made a ton of mistakes along the way. But I also learned a ton. And I’m still learning, but hopefully not making the same mistakes.
Almost all of my internships (and my first job at as an associate product manager at Evernote) came from these cold calls and cold emails. I had absolutely zero connections — and while I began building up my network over time, I very rarely asked friends to make introductions. I thought I could handle it myself.
And how did I do for myself?
I think I’ve come along so far rather nicely, even given hundreds of people have rejected me the past several years. Rejection still hurts, but I’ve learned to grow a thicker skin over time. And hey, look where I am today 🙂
…It’s so easy to look at someone’s accomplishments (with zero mention of all the rejections, struggles, blood sweat and tears involved) and get discouraged or wonder how so many good things happened to them so easily.
It’s only the successes that get mentioned. Never the struggles, failures and rejections… until after the fact, if at all.
Don’t get discouraged. Don’t compare yourself to others. Because you have no idea how many rejections they’ve had to go through in order to get to where they are today. All you see is the success of their hard work, but more importantly, persistence towards their goals. Look at James Dyson, JK Rowling, Sylvester Stallone and millions of others who are now famous (or maybe others not that famous at all, like me) after going through numerous rejections and failures.
Rejection is part of the process, and sometimes, good for you in steering you away from things that you thought you wanted, but actually would have been a bad fit for you. It can also be an excellent motivator for you to keep towards your goals.
Bottom line: Life is short. Take chances. Embrace rejection. Don’t fear it.
It’s all about your attitude, and as cliche as it sounds, about the journey rather than destination.
I wanted to share my story of rejections, and how that video on rejection and story has inspired me so much more than I can ever express.
To be nobody but yourself in a world which is doing its best, night and day, to make you everybody else means to fight the hardest battle which any human being can fight; and never stop fighting.
– e. e. cummings
VI. NEXT WEEK AND KEY LESSONS LEARNED
So next week will be making iterations on my MVP v3 based on last week’s user feedback and then continuing to do user testing with 5 more users this week. I hope to wrap things up to do a soft launch after iterating on that second round of user testing feedback.
Key lessons learned:
- For my purposes, a group of 5 users is a good number to do user testing.
- Try to think of your features in three buckets of prioritization: gamechanger, showstopper and distraction.
P.S. I’d love to meet you on Twitter here.
And if you enjoyed this post, please consider sharing it on FB or Twitter.