This time feels different: my updated idea validation process

This time feels different…

The first product I ever launched was called Ink - a contract generator and eSignature tool targeted at freelancers.

The whole thing actually started with the idea to build a Dribbble competitor, until a mentor convinced me I needed to actually talk to potential users first. It was my first brush with product discovery or idea validation.

We got a lot of great information out of those conversations, but we were young and knew nothing about marketing or sales, so we didn’t know what to do with it.

A few years later, I got inspired by Joel Gascoine’s landing page smoke test. I tried to replicate the same process for a new agency analytics tool called Albatross. I thought I had finally cracked it! We saw a 10% “conversion” rate on our fake signup flow and launched the MVP just one week later.

It flopped.

Then we got busy building an agency, where we watched client after client launch new products. I tried to learn from what worked for them, and what didn’t. At the same time I consumed everything I could on idea validation, product discovery, and building bootstrapped startups.

Then last year I was in position to take another swing.

I built another landing page with a fake signup flow, and reached out to my network to do some user interviews. I felt like I saw enough to build the MVP, and so I got to work. This time I was on my own, and my programming skills were rusty, so it took me 3 months rather than 1 week.

Once again… it flopped.

As I started to look for the next idea, I told myself I was going to do things the “right” way. I signed up for Rob Walling’s course, the SaaS Launchpad. I reread the Mom Test, and flipped through my product management books.

The results? It’s still early, but this time feels different.

Here’s what I did, what changed from my previous attempts, the tweaks I would make next time, and the limitations of product validation that may be unavoidable. I can’t guarantee this will work for you, and I know plenty of entrepreneurs who find success without doing anything like this, but this is what has worked best for me so far.

Note: for this post I’m focusing on validating an idea after I’ve done some initial market research and decided it’s worth pursuing further. In Rob Walling’s 2/20/200 framework, this is the 20 hour piece.

The process

User interviews

The primary focus of my validation process is still on user interviews. Beyond validation, you learn so much from talking to people that you can’t learn from async methods.

What does your ideal customer’s workflow look like today?
What existing tools are they using and paying for?
What language do they use to describe the market, their challenges, and their workflows?*
How do they respond to your messaging, pricing, or calls to action?

You also have the ability to ask follow up questions to dig deeper on important topics or pivot the conversation if you hear something interesting.

It’s still challenging to separate signal from noise, and easy to ask questions that can produce misleading answers (see the Mom Test), but there just isn’t any replacement for direct conversations.

For MetaMonster I sourced leads for conversations three ways:

My personal network
UserInterviews.com
Cold email with an offer to pay for their time

My network

The fastest and simplest way to get high quality prospects is by reaching out to your own network. This is a huge reason to build something in a space where you already have professional connections. For MetaMonster, I had enough overlap to schedule a few interviews.

While I’ve never been a professional SEO, I ran an agency for years and have a number of connections who are agency owners. Our first paying customer for MetaMonster was one of my closest friends, who uses it for his web design agency.

UserInterviews.com

UserInterviews is a site that recruits participants for interviews for you. It costs $49 per session plus $98 per session to use their B2B targeting and an incentive fee that’s given directly to the participant (typically a virtual gift card). They recommend $30-50 per interview for incentives if you are trying to attract a professional audience.

That adds up to almost $200 per session, but it’s worth it when you’re getting started. You can easily book and run 5-10 interviews in a week, which is way faster than you will be able to find and recruit prospects outside of your network on your own.

Cold email + offer to pay

The final method I used to find prospects was cold email. I followed Jason Cohen’s method, and offered to pay for people’s time. This worked well, given that my target audience are all consultants who are used to charging for their time. I had a few people offer to chat with me for free, and others quoted me anywhere from $50-$150 for 30 minutes of their time.

I used Apollo.io to source leads, Instantly.ai to send cold emails, and then paid people through whatever payment method they use for their clients.

I tried other offers and email copy, trying to see if I could get people to talk to me for free, but ultimately the only email that had any success before we had a product was the one with the payment offer (since we’ve launched I’ve had more success with other messaging).

From these conversations I was looking for a number of things:

Is there an existing workflow and/or market category our product can fit into?
What is the market’s willingness to pay? Do they already pay for tools? How many and how much do they pay?
How do they learn about new tools? How often do they try new tools?
Is there potential for them to be a customer? Are they open to an ongoing relationship?
How much do they trust and use AI in their current workflows? (MetaMonster is an AI tool)

I became convinced that SEOs were a good market for us and that MetaMonster was worth pursuing because:

We fit into an existing workflow (website optimization) that was time consuming and tedious with today’s tooling
SEOs are already paying for multiple tools, sometimes even paying for two tools that have overlapping functionality in order to get multiple perspectives
SEOs frequently try out new tools, and learn about new tools from professional blogs, influencers, and even cold email
When I asked what users would expect a tool like ours to cost, the numbers ranged from $50 per month to hundreds of dollars per month
Multiple prospects expressed interest in being early users with the expectation that the tool would be free to try and they would then have to pay once they were seeing value (and I quoted them a specific price)

In the future, I would spend more time digging into the details of their workflows and the purchase process. I think I got excited when I heard them independently talk about the problem I wanted to solve (writing metadata by hand) and could have learned a lot from digging into the specifics. I would be sure to ask them to walk me through the last time they bought a new tool (and actually started using it).

I would also try to take pre-payments rather than just asking for a general commitment to maybe pay in the future. For one, while the pricing feedback we got was helpful, people always think they’ll be more willing to pay than they actually are, and I think we could have tested our pricing a lot sooner by actually charging.

Maybe more importantly, taking payment means they have some skin in the game. We saw that most people stopped responding to us once a few weeks had passed and they got busy with other things, even if they seemed super engaged early on. And the early adopters who did sign up to use our product for free were less likely to thoroughly test it than the ones we eventually asked to put a credit card down and gave a time-limited trial.

At the same time, I have talked to friends who have taken pre-payments in the past, and still saw people who pre-payed never use the product. ¯\_(ツ)_/¯

Landing page

Once I’m feeling good about the viability of the product and the market from user interviews, and have a good set of notes to draw on for messaging, the next stage of my process is to set up a landing page.

This isn’t about product validation for me anymore, at this point I’m confident enough that the idea is worth pursuing to build the MVP.

This is a reversal of the process I’ve followed in the past, where I started with the landing page.

The value for me now is more in the process of writing the page, trying to distill down my initial positioning and messaging for the product, and then being able to test that messaging with prospects.

I used Carrd to create the initial landing page for MetaMonster. It took me a few hours to write the copy and then tweak one of their templates to fit with the minimalist branding I wanted to create.

I can then also validate that there is a reasonable amount of search traffic (aka latent interest) by running Google Ads, and begin to learn which terms appear to resonate more or less, what the costs are going to be to drive traffic for those terms, and what kinds of people are opting-in to your messaging (by looking at the types of emails coming through).

But again, this isn’t product validation, this is about testing positioning and marketing channels.

There is theoretically some value in starting to build a list of emails you can sell to when the product is ready. But not a single person on our list converted to a paying customer when we launched.

I have plenty of hypotheses for why that is (maybe I’m just a terrible email marketer) and what I would do different next time. But the point is - don’t overemphasize the value of this list vs an actual product signup.

The (probably) unavoidable limits of idea validation

Throughout all of this, it’s important to keep in mind that there are limits to how much you can learn without asking someone to pay for AND USE your product.

We had 300 people on our mailing list when we launched MetaMonster, only a handful have created accounts, and to-date none of them have converted to paying customers.

We had 9 people who told us in interviews that they were interested in being customers at prices ranging from $50 per month to low thousands per month, and to-date none of them have converted to paying customers.

We had 13 people who we gave free access to the alpha version of our product, and only 5 people have generated metadata for more than 10 pages. One of the people who I thought was a lock to be an early customer still hasn’t run a single generation. Another who seemed like a perfect fit and told me she would “use it daily,” has optimized 24 pages and hasn’t used the product in over a month.

It’s possible that asking people in interviews to pre-pa could have given us more accurate info, or that charging alpha users from day one would have given them more incentive to use the product. I certainly could have done a better job sending out updates and keeping users on our waiting list warm.

But I’ve talked to multiple founder friends who have tried it all, and results are mixed. One friend said that even when he got people to pre-pay, only a few of them actually used the product.

There are so many things that can change when it comes time for someone you talked to before the product was built to actually adopt your product.

They think the problem is more painful than it actually is
Your initial implementation is incomplete, buggy, or hard to use
They get pushback from other members of their team
People get busy and priorities shift

The reality is that there will always be limits to what you can learn before you have a product.

That doesn’t mean validation isn’t worthwhile, but there is no box you can check to say, “yes this is validated, it will work now.” Product development is a constant, ongoing process of trying to figure out where you are on the product/market fit spectrum, why, and then moving your way towards stronger fit.

The gap between what you can and can’t learn with product validation is where your product sense comes in.

So my recommendation is to do the work, talk to users, start testing your marketing channels, and then ship fast and repeat.