▲ 173 points
back
67 comments
In my experience it's better to pay attention to what your customers do, rather than what they say. Often peoples true goals aren't directly stated, or even known to themselves. If you asked people what type of store they want to shop in, very few would describe a walmart, and yet walmart seems to gain plenty of customers.
This sort of behavior is compounded when you throw in things such as distributed decision making. You will get piles of disjointed feedback, and almost none of it is informative about what actually drives adoption of your product.
My favorite book on customer development is "The Mom Test"[0], by a YCombinator alum. The name is terrible, but the content of that book is golden. He tells the terrifying story of talking to customers constantly as they wasted a million dollars building something the customers were not going to buy. There are various ways the customer pays you before the product is ready, and if they aren't paying you, they aren't really interested. Payment can be in the form of time (they meet with you), personal reputation (they recommend you to friends), company reputation (they arrange a meeting with their boss, or coworkers), etc.
[0] https://www.amazon.com/Mom-Test-customers-business-everyone-...
This is probably the book I recommend the most of any book, ever.
Memorable though ;)
>If you asked people what type of store they want to shop in, very few would describe a walmart, and yet walmart seems to gain plenty of customers.
I think a lot of people would say "a store that has what I want for a low price". It's the other characteristics of walmart that put people off, not the core value proposition.
You're right, they'd say that. They wouldn't mention "within 10 minutes drive of my house" or "has plenty of parking" or anything about the return policy most likely. And yet those almost certainly factor into where they shop.
Oh for sure. It's tough in the services business here though where a lot of us find ourselves in when we're building products on the side. We have to build on what our customers say a lot. One thing that helps over here at Rockstar is we simply encourage and orchestrate things like bringing in actual users to play with mockups and such.
And some things are really hard to just observe without asking. In the interviews I mentioned, we really wanted to get into the psychological state and emotions people felt while they were shopping to understand what our brand even meant in the marketplace. It's impossible to observe that well unless we get into paying attention to what these folks say in interviews. And then we find ourselves having to interrogate those responses like I mentioned.
What are your favorite ways of observing customers? I've been a fan of Heap Analytics. Any others out there catch your eye?
That's number one. Also never, ever believe your salespeople when they tell you what customers want. Take it all in but make sure you talk to some customers directly.
I have had it so many times that we implemented some convoluted only because the customer and salespeople misinterpreted some technology or didn't know about other ways to do what they need.
It depends what you're optimizing for. Sometimes when Walmart's customer satisfaction goes up, their revenues go down. https://web.archive.org/web/20130524132122/http://dailyartif...
and even less are going to describe Amazon.
One hack we've employed with our customers in interviews and through surveys: have them rank/prioritize our current features, and consider/rank a set of potential new features. It's much easier for someone to order a list (reaction), than to provide an explanation (creation). Not a substitute for a full interview, but a quick way to determine what your customer values.
How do you control for people's abstract desires not mapping well to what they actually want?
Example to illustrate my point: Focus groups over the past 15 years or so have consistently told car mfrs that consumers want larger cars - larger cars are assumed to be more luxurious and associated with financial success, and thus are desirable. The end result is that previous small, manoeuvrable, city cars are now bigger and harder to park. Consumers or small cars are generally unhappy with this trend, as they wanted a small car for the ease of parking etc, but focus groups are frequently insufficiently well tuned to pick up up on this nuance.
My experience in this industry over many years is that users often don't have a clear idea of what they want, and in the cases when they do a non-trivial amount of time it's a so tightly tethered to one special-snowflake usecase that virtually noone else finds value in it.
An important effect of asking people to rank choices is that it forces people to consider costs. An increase in the rank of one item inherently decreases the rank of another.
This is really interesting. Have results tended to be relatively uniform across customers or do you end up having to break a lot of hearts when you ultimately prioritize features on your roadmap? Do you have them rank UX-driven changes (i.e. reframing existing features) in addition to entirely new features?
One thing I've found that works is asking them to write down free-form what they value, and then give them the list you've prepared and then ask them to rank.
Unless you're detached from your customer, they should be in similar ballparks, but getting the words they actually use out of them, and then ranking what you had in mind can be powerful for clarifying their thought process (and gives your marketing team the messaging from the horse's mouth).
Yeah I have also found that asking cusomers to rank or sort features using a question similar to this one:
https://www.surveyking.com/help/ranking-question.php
It’s a very simple but effective way to gauge levels of reaction without losing the interest of the audience.
Very interesting. What tool/s do you use to gather this feedback and get the aggregated view across all clients? Do you weigh all clients the same?
My father worked for a large software company for many years. I remember him telling me how they successfully responded to a Request For Proposals with a much smaller proposal that said, in essence, "You don't know what you're actually looking for; pay us $ to help figure that out before you pay $$$ to build it."
I do this when I consult. I call it a roadmapping session. It's massively successful. The customer pays a little bit of $$ relative to the total spend to get what they want built.
In exchange they get:
1. A doc outlining what they want built and how it'll be built. This is the key deliverable.
2. To work with me and see if it's a good fit
3. To pay a fraction of the price to de-risk a larger engagement
4. To see if I can deliver on what I said I'd build for them
And I get:
1. To get paid to discover the bounds of their project (vs. writing a proposal for free)
2. To see if they're a good client (pays on time, works well together, etc)
It's a huge win for everybody.
Oh I have a good story about that! Healthcare software is rife with these sorts of things.
We had a potential client come in saying they had the toughest requirements; terabytes of data would be collected, we'll need PhD data scientists to write very complex algorithms, huge servers to handle the incredible load and uptime needed, full suite of mobile and desktop apps, blah blah blah.
We analyze his requirements a bit and it turns out he needs actually just needs a survey with 5 questions, none of which collected personal information. And he treats 20 patients a year. We call him in, set him up a Google Form in front of him, and sent him on his way. Probably saved the Canadian healthcare system millions on that one.
Looking back on this experience, I think some people are incredibly ego-driven, so the prospect of doing something off-the-shelf or for free doesn't enter their conscience.
I love that. Exactly. Any more you can share about that story? I imagine customer loyalty or repeat projects after that proposal went up too? There's another post I'm kicking around about how a service vendor I did business with recently basically proposed what I intended. Charged up to my budget. And the project was a bust. And I'll never recommend them to anyone. Tracing back the steps though, if that vendor had said "Look. What you're trying to do might not work. Let's just try something smaller for much less than your budget." I would have gone for it. Then even with failure I would have kept trying with them, and recommending them to others. Too many of us agencies and freelancers try to optimize for the $ on the table now, and we put the long term relationship in jeopardy (the one often worth much more $$$)
We do this as well for our "DevOps-as-a-Service" consulting. We have a phase I discovery session before the build session where we map out different potential solutions and the cost / benefit of each one.
An unstated problem of trying to gauge customer (or prospective customer experience) is orthogonal to this article: people are very reluctant to tell you to your face that your baby's ugly.
I have had the experience of launching a product where people gave the idea rave reviews, were willing to pilot it (and the pilots met their success criteria) but garnered almost no sales. The fact was though it addressed what CxOs claimed was one of their top 3 problems, they just didn't want to do what was necessary to actually deal with it. (I think if we had actually had the problem ourselves we would have known this).
My experience with my current company could be considered the opposite of this. People are willing to pay us large amounts of money (surprisingly so) because they need a solution for this particular problem and nothing exists, but they hate what we have built and constantly complain / write angry emails offering feedback of all varieties. What they don't do is leave, despite not being locked in. Strange.
omg yes! I think about dining out. Every time I've had a terrible meal or experience:
Host: How is everything? Me: Great
I just want them to go away and just not return or tell anyone to come here.
It's true. It's really hard to pull out negative feedback. Hence you're absolutely right, being your own customer definitely raises your chance for building the right thing here.
>people are very reluctant to tell you to your face that your baby's ugly
I like your phrasing a lot more but there's a term for this: https://en.wikipedia.org/wiki/Social_desirability_bias
Social desirability bias comes up a lot in UX research and how to engage with users. In theory different methods exacerbate or reduce the effects (e.g. focus groups are highly impacted, individual surveys less so).
> An unstated problem of trying to gauge customer (or prospective customer experience) is orthogonal to this article: people are very reluctant to tell you to your face that your baby's ugly.
That's so true. IIRC this lesson is imparted early and often in "lean startup" resources: Testing a product/pricing hypothesis requires that customers "get out their wallet" — that they pony up actual money, or at least click a button where they believe they'll be required to do so.
This form of user interviews with existing customers also inoculates your customer against marketing from competitors, since they are now more invested in defending their decision to buy your product.
I heard that similar tactic on some podcast by a services guy. I can't remember the actual field, let's say "lose weight" services. They ask the new client "How committed are you to losing weight?" Let's say the client says "4 out of 5". You might think you should ask: "Why aren't you a 5?" But the ninja move is to ask "But why aren't you a 3?" This forces the client to tap into their own self-motivation (which is all there is, after all) and they start to convince themselves why they are committed to the results.
Ah that's interesting. Hadn't thought of it that way before. Have you seen any more research about that topic? I'd love to see more.
This reminds me of the classic Monty Python sketch called "Argument Clinic".
no it doesn't
Yes, and apply the same philosophy to your product manager, etc. as a developer. Developers tend to assume that the manager has carefully considered the cost of what they ask for, and is certain that the result is worth it. But this usually isn't true. Managers tend to ask for more than the developers can deliver. They do not always remember to make clear to employees which requests are essential vs. "nice to have". I think this is often an honest mistake - the manager does not realize they are failing to communicate their priorities.
If the cost/benefit seems fishy, the developer should ask the manager for more justification. A lot of developers don't realize they can push back.
Not just that they can push back, but that the business, product or service, and developer happiness is better if the developer and manager partner on understanding the business problem and solving it. This is in contrast to the developer just "implementing to spec" and then being surprised when the end user is bummed that "this doesn't work the way I thought it would".
We are experts in our domain of software development and do no one favors when we hide it and just do what we're told.
(That isn't to say that the developer has the only opinion that matters, just that they are a stakeholder with expertise and should act like it.)
The crux of this is:
Giving the customer what they ask for != solving the customers problem.
Just writing down my train of thoughts here. This article made me think of a machine learning-like approach of learning the customer needs. You want to find out the true distribution of them. Apparently, the more clearly you can identify the needs, the more win for your products. But in some areas, the customers are "noisy" and the boundary (of what's okay and what's not) is not clear. So you want to try various approaches to figure out the shape of what the "needs" look like.
In some sense, brainstorming (without criticizing the ideas) is a bit like generative methods. You only want to find the center of distribution. But sometimes you want to know the exact boundary. In that case, you need to employ a discriminative way; to try both sides of the boundary to define the better shape of it.
Always. I’ve been practicing this for 10 years, most of which I’ve been working as a consultant building custom software for internal business processes, as well as customer-facing software for clients’ users. My typical process looks something like this when a feature/change request or random idea is asked for—which almost always comes in along the lines of, “Can we get X added in that does Y and looks like Z?”:
1. I never say yes to any request immediately. I always tell a client, “Anything is possible that doesn’t violate the laws of physics, but let’s dig into this more.”
2. Ask what they’re trying to accomplish. What problem are they trying to solve? What mistake are they trying to prevent? What is the end goal? I ask questions until I can explain back to the customer what they’re really wanting to do and why.
3. I then push back on why I think we should not do what they’re asking for in the way they’re asking for it. Not in an asshole way, but I do challenge their request, its utility, the value it will bring the business/customers/employees, etc. When doing this, I always spend time on educating them on the tech behind the scenes, potential pitfalls, how adding something (especially if it’s visible to people) will make it nearly impossible to ever remove it once users get used to it, when they’re asking for something that would be more effort the way they’re asking for it to be done than it’s worth when it’s trying to solve for a rare edge case, etc.
4. After explaining the problems with their idea as given by non-experts, I start suggesting other ways we could accomplish their goals--using a simpler UX, or even no UX at all--relying on the ability to automate things if we have enough information, hiding all the complexities of a given business process behind a single button once we have the right info to intelligently take action.
5. I give a couple of recommendations for potential ways forward that solve the real problem in a way I’ll enjoy building it out. I spend time explaining what I see as the benefits & trade-offs with my suggestions for solving their problems, as well as how long the options should take. I try to always include at least one option that is as simple and quick-to-build as possible (internal MVP approach), and one option that has more bells and whistles (when relevant/appropriate). Then I let them make the choice.
Following this pattern has pretty much never failed me. What feels best about it is when I see clients actually learn how their software works when I’m working for them. I love it when they remember the discussions we’ve had, internalized it, and recall it when we talk 6 months later about their new idea. Over time, their ideas improve because their understanding of how their software works improves. They also become increasingly invested in our working relationship as their trust in my concern for solving their problems—and not just doing their bidding—increases.
Never shy away from challenging your customers’ ideas—but always do it in a respectful manner that gets to the heart of their real problems and educates them along the way. They’ll appreciate it, and will keep coming to you for more. I don’t think this is unique to being a consultant, either—the same sort of process can be followed with direct users of your own product.
This is pretty much the pattern I follow. People like to come up with solutions and so they ask - can you do this? The answer is of course ‘yes’, but it’s not necessarily the correct solution to the unstated problem.
I work scaling support teams on a daily basis and I couldn't agree more with this.
Beyond the product itself, there's also the question about the relationship with your team, which is often very important. In that regard, it's even worse, and many customers will try to be very nice and say that your team was very helpful, omitting some redflags of a bad customer support experience.
I remember a saying from a friend as follows:
"A relationship with customer is like a marriage.
If everything is too perfect, there are no conflicts or issues, and you just love each other....
There is definitely something wrong. "
It's imperative to 'open them up' and find out what's really happening before it's too late.
Although browbeating does have a lot of value, especially when there are already some bounds on what the problem is and how it can be solved with your product, you will hit a point where your customers say, "I don't know" and mean it. And that's when you have to start to treat feedback more holistically and use the customer as a final validation of an internal model. [1]
Some companies I’ve worked with aren’t able to specify their requirement. Sometimes it’s better to say to them - let’s build a product that adheres to your stated requirements, and then you can use it. This allows them to identify all the parts of the business process they hadn’t been aware of or anticipated and provides a much better basis for creating a clear specification that is built on a shared understanding of the actual business processes and requirements.
Two problems:
1.) If you have to pay people to give feedback, you're doing something very wrong.
2.) If you interrogate a customer who cares enough to give you candid feedback, your odds of ever getting candid feedback from that customer drop to near zero.
The quest for meaningful feedback cannot overwhelm the need to serve customers and treat them well.
> If you have to pay people to give feedback, you're doing something very wrong.
Why? This is a more unbiased way of getting feedback than letting your most vocal users tell you about their problems with your product. Paying users for feedback is more common in B2C apps but happens quite a bit in B2B also.
I have found that saying "why are you asking for X...can you tell me what you are trying to achive with X?" works wonders as a starting point. Make it about the outcome and not the how is a big part of moving forward and being successful.
I'm just reading Talking to humans and this fits nicely with it. Highly recommended
Osborn's "Brainstorming" methodology had 2 main tenants
1. It should be tenets. A tenant is "a person who occupies land or property rented from a landlord". A tenet is a "principle, opinion, or dogma maintained as true by a person, sect, school; a thing held to be true"[0]. I've seen this mistake a couple of times before on programmer blogs.
2. I think he means method. It seems methodology is starting to take over from method in a lot of areas, maybe because it's longer and sounds more impressive and scientific. No-one wants to talk about their method when they can talk about their methodology. But in scientific papers, the methodology is the part where they explain the reasons why they're using the methods they're using. The two words mean two different things.[1] Let's keep it that way!
[0] The Online Etymology Dictionary goes on: early 15c., from Latin tenet he holds, third person singular present indicative of tenere to hold, grasp, keep, have possession, maintain, also reach, gain, acquire, obtain; hold back, repress, restrain; figuratively hold in mind, take in, understand https://www.etymonline.com/word/tenet
Incidentally, I didn't realize until I started learning Spanish that "ten" or "tain" in English words means to have. Spanish inherited tenere as tener, to have, and adds prefixes to make a lot of other verbs, e.g. abstener, contener, detener, mantener, obtener, retener, sostener, all of which mean something like what they do in English - to abstain, contain, detain, maintain, obtain, retain, sustain - various forms of 'having' (or not having). "Ten" is in words like tenacious - from Latin tenax, holding fast, clinging, and tenable - capable of being held/maintained.
[1] "Etymologically, methodology refers to the study of methods. Thus the use of methodology as a synonym for methods (or other simple terms such as means, technique, or procedure) is proscribed as both inaccurate and pretentious." https://en.wiktionary.org/wiki/methodology
Isn't this plain ole rational skepticism?
Us: Why did you choose to buy our product?
User: I liked it over your competition.
Us: Great. What was better about us?
User: I don't know. I just liked it.
Ugh. If we have a week of conversations like this,
we'll end up with nothing.