Product Discovery methods

This post is based on Inspired, by Marty Cagan, with my personal comments and examples.

One of the essential skills (you can read more on PM skills here) of a Product Manager is to know and use various product discovery testing techniques in order to evaluate and assess 4 primary risks. The product discovery phase is conducted before any actual development will start. The risks I am talking about are the following:

• usability risk — can the user figure out how to use this?
• value risk — will the user or customer choose to use or buy this?
• feasibility risk — can we build this?
• viability risk — is this solution viable for our business?

"Product discovery is a technique that is used to quickly separate the good ideas from the bad as we work to try to solve the business problems assigned to us". Marty Cagan

Usability testing

This technique is the oldest one and have been existing for many years.

There are a lot of online tools that can help you to facilitate these tests (for example usertesting.com), or you can do all the user test phases yourself.

In order to run a user test, you need:

1. Audience — you have to recruit users to participate in the testing. You can use your dataset of existing users, you can advertise participation in social media, you can find users in professional events/conferences, or you can use online platforms that already have an audience, from which you can pick a user target profile that you need.
2. Prepare the test — you can ask users to test feature that is already available online, or you can ask users to test a high-fidelity prototype. For both cases you have to prepare a set of tasks, that you want a user to test — a flow, a usecase, or some set of questions to be answered/found.
3. Performing the test — test can be done onsite, or online via video. You would like to see a screen of a user — how he navigates on that, where he clicks, how he scrolls. The user should also make oral comments on what he sees, what he expects to see, and what is his reaction and thoughts. There are three important cases you are looking for: (1) the user got through the task with no problem at all and no help; (2) the user struggled and moaned a bit, but eventually got through it;(3) he got so frustrated he gave up.
4. Participating in the test — it is important that the Product Manager, designer, and engineer from the team — all of them will participate in user interviews. The magic often happens when engineer is present, so I also encourage that participation whenever possible. I remember the tests that we were performing with my product team and how frustrated the engineer once was — he was so angry at the customer who did not understand the navigation and was clicking on the wrong button over and over again. After the second user fell into the same loop, the engineer got the point — if the flow was logical only to the engineer, and not to the user, then there is something wrong with flow logic :)
5. Summarize the test — after the tests are done, some summary should be written down, including the hypothesis, link to the object under test, size of the user group, and findings from the exercise. It is great is this summary will be shared with another part of the product team and with other product teams — the more we know how our customer behaves, the better we build different aspects of our product.

Value Testing

Customers will buy our product only if they perceive real value.

Having the same features as our competitors have is not enough for a client to do a switch and become our client. the product should be substantially better in order to motivate clients to take the whole hassle of moving and switching from their old solution.

Demand Testing is one of the categories of Value testing. There are some easily implementable techniques to test, is there any actual demand for e new feature, or product, or price plan? Before you will actually be spending time and effort on developing those, try to estimate the demand by fake door demand test (landing page demand test).

The idea is that you put a button or menu item into the user experience flow exactly where we believe it should be on the screen. When the user clicks on the button, instead of leading to the new feature/product client will see a special page, that explains the experiment. The explanation will say that company is testing the demand for such feature/product and seeking customers to talk about this. You can also offer clients to sign up and become the first user-testing group of new feature/product in the future.

With the proper analytics installed on the explanation page, you will be able to test the demand for feature/product as well as to collect a potential list of usertesting reference group.

Qualitative Value Testing is a test that is performed after the user has concluded the user testing of the product flow (described above). The questions should be asked to understand if the user would be willing to pay for a product, even if you have no intention of charging them for it. You can also ask a series of questions to understand if the user would be willing to pay their reputation — how likely they’d be to recommend the product to their friends or on social media.

Quantitative Value Testing is all about collecting evidence from a live usage environment for our prototype. The best way to do that is to use discovery A/B testing, where 1% of all users will see instead of regular product flow live-data prototype. Prototype live usage should be closely monitored and measured, in order to draw conclusions based on the hard numeric evidence on how clients feel about the new feature.

Feasibility Testing

While assessing feasibility, developers try to answer the questions regarding the complexity of implementation: how the solution should be built? what skills are needed for this? Is there needed architectural changes to be applied? Will the performance be acceptable? Do we have necessary infrastructure to test and run?

Technology is evolving very fast — things we did not imagine could be doable 10 years ago, are everyday life now.

Another aspect of ever-changing technology is, that engineers have to have tie to learn and investigate new technology. There cannot be sprint-after sprint, task after task without a time to lean, test and discover new versions, new ideas, new technologies.

Business Viability Testing

It is not enough for a product to be loved by clients and implementable by engineers. The solution must also work for your business.

Building a business is always hard. You must have a business model that’s viable. The costs to produce, market and sell your product must be sufficiently less than the revenue your product generates. You must operate within the laws of the countries you sell in. You must hold up your end of business agreements and partnerships. Your product must fit within the brand promise of your company’s other offerings.

This is what really meant by being the CEO of the product.

It is a tendency to forget about future maintenance costs for a product, when testing business viability — the cost of future customer support, regulatory and management reporting, back-office automation of maintenance processes, unhappy customer flows — they all will require a lot of effort, time and money to solve. Unfortunately these expenses are often not taken into account when analysing business viability for a new product of feature.

Conclusion

Creating new product (that customers see valuable and that bring your company profits) is a hard exercise that need preparation and analyses. It should be always a balance between agile fast movement and deep thorough analyses. I hope you will find it :)