The great Jim Collins has this line I really like where he says fire bullets then cannonballs. “The right big things are the things you’ve empirically validated. So, you fire bullets, you validate, then you go big — bullets, then cannonballs — it’s both.”
Too often — something I see in me as much as I see in founders — when trying something new, we bottle it up. We charge the entropy of our creativity. Waiting to release it all at one big moment. A cannonball. No one else should or needs to know know. Sometimes it’s a fear of someone else stealing your idea. Sometimes, well, speaking more for myself, I just like surprises. I love the mystique. And on the slim chance you’re right, albeit rare, then awesome. But 999 out of 1000 times, you’re likely not. At least not in the first try.
I’m forgetting and also can’t seem to find the attribution. But I read somewhere that the only difference between vision and a hallucination is that others can see it. You see… the greatest YouTubers test their ideas with test audiences several times. In fact, they even test their video titles with select audiences a number of times before launching. (Instagram even added the ability to do it at scale for creators too.) Reporters do too with their headlines. Legendary investor Mike Maples at Floodgate once said, “90% of our exit profits have come from pivots.” ONSET Ventures also found in its research1 that founded the institution back in 1984 (prescient, I know) that there is a 90% correlation between success and the company changing its original business model.
All to say, one’s first idea may not always be the best and final idea. So, test things. With small audiences. With trusted confidants.
And while I may not do this all the time, with my bigger blogposts (like this, this, and this), I always run it by co-conspirators, subject-matter experts, lawyers, writers, bloggers, and people who love reading fine print. And sometimes the final product may not look like the one I initially intended, which will be true for an upcoming bigger blogpost. For events, like one I recently worked with the team at Alchemist on — redefining what in-person Demo Days look like at accelerators, we tested the idea with 20 other investors and iterated on their feedback before launching on January 30th this year. And still is not even close to its final evolution.
As Reid Hoffman once said, “If you are not embarrassed by the first version of your product, you’ve launched too late.”
Now that it’s out there…
One of the greatest Joker lines in The Dark Knight is: “Trust no one, salt and sugar look the same.”
It’s true. Whether people like something or not, they’ll always tell you things were good. It’s the equivalent of when one goes to a restaurant, orders something that’s a bit saltier than one’s liking, but when the server comes by to ask, “How is everything?”, most people respond with “Everything’s fine.” Or “good.”
You’re not going to get the real answer out of people oftentimes. Unless people really do love or hate something you did passionately. So… you must hunt for them. You must lure out the answers. You need to force people to take sides. There can be and shouldn’t be middle ground. If there are, that means they don’t like it.
Maybe it’s in the form of the NPS question. On a scale of 1-10, how likely would you recommend this product to a friend? And you cannot pick 7.
In the event space, I’ve come to like a new question. If I invited you to this event the week of, would you cancel plans to make this event? And to add more nuance, what kinds of events would you cancel to be here? What kinds of events would you not cancel?
Sometimes it helps to seed examples on a spectrum (although I try not to lead the witness here). Would you cancel a honeymoon? Or would you cancel going to another investor/founder happy hour? What about an AGM (annual general meeting, annual conference in VC talk)? What about a vacation?
As Joker said, salt and sugar look the same. So you have to taste it. Looking from afar won’t help. And if you want to iterate and improve, you need what people really think. I’d rather have people hate or dislike something I’ve created than have a lukewarm or worse, a “good” reaction.
In a way, if you’re not getting enough of an auto-immune response from the crowd, and the antibodies don’t start kicking in (aka the naysayers), you’re not really doing something new.
1 FYI, the research link redirects to its HBS case study, not the original research. Couldn’t find the latter unfortunately. But the point stands.
Stay up to date with the weekly cup of cognitive adventures inside venture capital and startups, as well as cataloging the history of tomorrow through the bookmarks of yesterday!
The views expressed on this blogpost are for informational purposes only. None of the views expressed herein constitute legal, investment, business, or tax advice. Any allusions or references to funds or companies are for illustrative purposes only, and should not be relied upon as investment recommendations. Consult a professional investment advisor prior to making any investment decisions.
As I am co-leading a VC fellowship with DECODE (and here’s another shameless plug), a few fellows asked me if I had a repository of questions to ask founders. Unfortunately, I didn’t. But it got me thinking.
There’s a certain element of “Gotcha!” when an investor asks a founder a question they don’t expect. A question out of left field that tests how well the founders know their product, team or market. In a way, that’s the sadist inside of me. But it’s not my job, nor the job of any investor, to force founders to stumble. It’s my job to help founders change the world for the better. By reducing friction and barriers to entry where I can, but still preparing them as best as I can for the challenges to come.
I’m going to spare you the usual questions you can find via a quick Google search, like:
What is your product? And who is your target audience?
How big is your market? What is your CAGR?
What is your traction so far?
How are you making money? What is your revenue model?
And many more where those come from.
Below are the nine questions I find the most insightful answers to. As well as my rationale behind each. Some are tried and true. Others reframe the perspective, but better help me reach a conclusion. I do want to note that the below questions are described in compartmentalized incidents, so your mileage may vary.
Here’s to forcing myself into obsolescence, but hopefully, empowering the founders reading this humble blog of mine to go further and faster.
The questions
I categorize each of the below questions into three categories:
The market (Why Now)
The product (Why This)
And, the team (Why You)
Together, they form my NTY thesis. The three letters ordered in such a way that it helps me recall my own thesis, in an unfortunate case of Alzheimer’s.
Why Now
What are your competitors doing right?
This is the lesser-known cousin of “What are your product’s differentiators?” and “Why and how do you offer a better solution than your competitors?”. Founders are usually prepared to answer both of the above questions. I love this question because it tests for market awareness. Too often are founders trapped in the narratives they create from their reality distortion fields. If you really understand your market, you’ll know where your weaknesses are, as well as where your competitors’ strengths are.
There have been a few times I’ve asked this question to founders, and they’d have an “A-ha!” moment when replying. “My competitors are killing it in X and Y-… Oh wait, Y is our value proposition. Maybe I should be prioritizing our company’s resources for Z.”
Why is now the perfect time for your product to enter the market?
As great as some ideas are, if the market isn’t ripe for disruption, there’s really no business to be made here… at least, not yet. What are the underlying political, technological, socio-economical trends that can catapult this idea into mass adoption?
For Uber, it was the smartphone and GPS. For WordPress and Squarespace, it was the dotcom boom. And, for Shopify, it was the gig economy. For many others, it could be user habits coming out of this pandemic that may have started during this black swan event, but will only proliferate in the future. As Winston Churchill once said, “Never let a good crisis go to waste.”
A great way to show this is with numbers. Especially your own product’s adoption and retention metrics. Numbers don’t lie.
What did your customers do/use before your product?
What are the incumbent solutions? Have those solutions become habitual practices already? How much time did/do they spend on such problems? What are your incumbents’ NPS scores? In answering the above questions, you’re measuring indirectly how willing they are to pay for such a product. If at all. Is it a need or a nice-to-have? A 10x better solution on a hypothetical problem won’t motivate anyone to pay for it. A 10x on an existing solution means there’s money to be made.
Before we can paint the picture of a Hawaiian paradise, there must have been several formative volcanic eruptions. It’s rare for companies to create new habits where there weren’t any before, or at least a breadcrumb trail that might lead to “new” habits. As Mark Twain says, “History doesn’t repeat itself, but it often rhymes.”
Why This
What does product-market fit look like to you?
Most founders I talk to are pre-product-market fit (PMF). The funny thing about PMF is that when you don’t have it, you know. People aren’t sticking around, and retention falls. Deals fall through. You feel you’re constantly trying to force the product into your users’ hands. It feels as if you’re the only person/team in the world who believes in your vision.
On the flip side, when you do have PMF, you also know it. Users are downloading your product left and right. People can’t stop using and talking about you. Reporters are calling in. Bigger players want to acquire you. The market pulls you. As Marc Andreessen, the namesake for a16z, wrote, “the market pulls product out of the startup.”
The problem is it’s often hard to define that cliff when pre- becomes post-PMF. While PMF is an art, it is also a science. Through this question, I try to figure out what metrics they are using to track their growth, and inevitably what could be the pull that draws customers in. What metric(s) are you optimizing for? I wouldn’t go for anything more than 2-3 metrics. If you’re focusing on everything, you’re focusing on nothing. And of these 1-3 metrics, what benchmark are you looking at that will illustrate PMF to you?
For example, Rahul Vohra of Superhuman defines PMF with a fresh take on the NPS score, which he borrows from Sean Ellis. In feedback forms, his team asks: “How would you feel if you could no longer use the product?” Users would have three choices: “very disappointed”, “somewhat disappointed”, and “not disappointed”. If 40% or more of the users said “very disappointed”, then you’ve got your PMF.
Founders don’t have to be 100% accurate in their forecasts. But you have to be able to explain why and how you are measuring these metrics. As well as how fluctuations in these metrics describe user habits. If founders are starting from first principles and measuring their value metric(s), they’ll have their priorities down for execution. Can you connect quantitative and qualitative data to tell a compelling narrative? How does your ability to recognize patterns rank against the best founders I’ve met?
If in 18 months, this product fails. What is the most likely reason why?
This isn’t exactly an original one. I don’t remember exactly where I stumbled across this question, but I remember it clicking right away. There are a million and one risks in starting a business. But as a founder, your greatest weakness is your distraction – a line in which the attribution goes to Tim Ferriss. Knowing how to prioritize your time and your resources is one of the greatest superpowers you can have. Not all risks are made equal.
As Alex Soktold me a while back, “You can’t win in the first quarter, but you can lose in the first quarter.” The inability to prioritize has been and will continue to be one of the key reasons a startup folds. Sometimes, I also walk down the second and third most likely reason as well, just to build some context and see if there are direct parallels as to what the potential investment will be used for.
On the flip side, one of my favorite follow-ups is: If in 18 months, this product wildly succeeds. What were its greatest contributing factors?
Similar to the former assessing the biggest threats to the business, the latter assesses the greatest strengths and opportunities of this business. Is there something here that I missed from just reading the pitch deck?
What has been some of the customer feedback? And when did you last iterate on them?
I’m zeroing in on two world-class traits:
Open-mindedness and a willingness to iterate based on your market’s feedback. As I mentioned earlier with Marc Andreessen’s line, “the market pulls product out of the startup.” Your product is rarely ever perfect from the get-go, but is an evolving beast that becomes more robust the better you can address your customer’s needs.
Product velocity. How fast are your iteration cycles? The shorter and faster the feedback loop the better. One of the greatest strengths to any startup is its speed. Your incumbents are juggernauts. They’ll need a massive push for them to even get the ball rolling. And almost all will be quite risk-averse. They won’t jump until they see where they can land. Use that to your advantage. Can you reach critical mass and product love before your incumbents double down with their seemingly endless supply of resources?
Why You
What do you know that everyone else doesn’t know, is underestimating, or is overlooking?
Are you a critical thinker? Do you have contrarian viewpoints that make sense? Here, I’m betting on the non-consensus – the non-obvious. While it’s usually too early to tell if it’s right or not, I love founders who break down how they arrived at that conclusion. But if it’s already commonly accepted wisdom, while they may be right, it may be too late to make a meaningful financial return from that insight.
But if you do have something contrarian, how did you learn that? I’m not looking for X years of experience, while that would be nice, but not necessary. What I’m looking for is how deep founders have gone into the idea maze and what goodies they’ve emerged with.
Why did you start this business?
Here, unsurprisingly, I’m looking for two traits:
Your motivation. I’m measuring not just for passion, but for obsession and the likelihood of long-term grit. In other words, if there is founder-market fit. Do you have a chip on your shoulder? What are you trying to prove? And to whom? Do you have any regrets that you’re looking to undo?
Most people underestimate how bad it’s going to get, while overestimating the upside. The latter is fine since you are manifesting the upside that the wider population does not see yet. But when the going gets tough, you need something to that’ll still give you a line of sight to the light at the end of the tunnel. Selfless motivations keep you going on your best days. Selfish motivations keep you going on your worst days.
Your ability to tell stories. Before I even attempt to be sold by your product or your market, I want to be sold on you. I want to be your biggest champion, but I need a reason to believe in the product of you. You are the product I’m investing in. You’re constantly going to be selling – to customers, to potential hires, and to investors. As the leader of a business, you’re going to be the first and most important salesperson of the business.
What do you and your co-founders fundamentally disagree on?
No matter how similar you and your co-founders are, you all aren’t the same person. While many of your priorities will align, not all will. My greatest fear is when founders say they’ve never disagreed (because they agree on everything). To me, that sounds like a fragile relationship. Or a ticking time bomb. You might not have disagreed yet, but having a mental calculus of how you’ll reach a conclusion is important for your sanity, as well as the that of your team members. Do you default on the pecking order? Does the largest stakeholder in the project get the final say after listening to everyone’s thoughts?
Co-founder and CEO of Twilio, Jeff Lawson, once said: “If your exec team isn’t arguing, you’re not prioritizing.”
I find First Round’s recent interview with Dennis Yu, Chime’s VP of Program Management, useful. While his advice centers around high-impact managers, it’s equally as prescient for founding teams. Provide an onboarding guide to your co-founders as to what kind of person are you, as well as what kind of manager/leader you are. What does your work style look like? What motivates you? As well as, what are your values and expectations for the company? What feedback are you working through right now?
In closing
Whether you’re a founder or investor, I hope these questions and their respective rationale serve as insightful for you as they did for me. Godspeed!
Stay up to date with the weekly cup of cognitive adventures inside venture capital and startups, as well as cataloging the history of tomorrow through the bookmarks of yesterday!
I’ve written about product-market fit on numerous occasions including in the context of metrics, pricing, PMF mindsets, just to name a few. And one of the leading ways to measure PMF is still NPS – the net promoter score. The question: On a scale of one to ten, how likely would you recommend this product to a friend?
As investors, while a lagging indicator, it’s a metric we expect founders to have their finger always on the pulse for their customers. Yet how often do investors measure their own NPS? How likely would you, the founder, recommend this fund/firm/partner(s) to your founder friend(s)?
Let’s look for a second from the investor side of the table…
Mike Maples Jr. of Floodgate pioneered the saying, “Your fund size is your strategy.” Your fund size determines your check size and what’s the minimum you need to return. For example, if you have a $10M pre-seed fund, you might be writing 20 $250K checks and have a 1:1 reserve ratio (aka 50% of your funds are for follow-on investments, like exercising your pro rata or round extensions). Equally so, to have a great multiple on invested capital (MOIC) of 5x, you need to return $50M. So if you have a 10% ownership target, you’re investing in companies valued around $2.5M. If two of your companies exit at $200M acquisition, you return $20M each, effectively quadrupling your fund. You only need a couple more exits to make that 5x for your LPs. And that’s discounting dilution.
On the flip side, if you have a $100M fund with a $2-3M check size and a 20% ownership target, you’re investing in $10-15M companies. Let’s say your shares dilute down to 10% by the time of a company’s exit. If they exit at unicorn status, aka $1B, you’ve only returned your fund. Nothing more, nothing less. Meaning you’ll have to chase either bigger exits, or more unicorns. But that’s hard to do. Even one of the best in the industry, Sequoia, has around a 5% unicorn rate. Or in other words, of every 20 companies Sequoia invests in, one is a unicorn. And that means they have really good deal flow. Y Combinator and SV Angel, who have a different fund strategy from Sequoia, sitting upstream, have around 1%.
Up to a $20m fund, there are are many ways to do 3x+
Up to a $60m fund, there are several ways
After $150m fund, there is probably 1 way
— Jason ✨Be Kind✨ Lemkin 🇮🇱 (@jasonlk) April 5, 2017
Why does a VC’s fund strategy matter to you as the founder?
A fund with a heavily diversified portfolio, like an angel’s or accelerator’s or participating investors (as opposed to leads), means they have less time and resources to allocate to each portfolio startup. The greater the portfolio size, the less help on average each startup team will get. That’s not to say you shouldn’t seek funding from funds with large AUMs (assets under management). One example is if you have an extremely passionate champion of your space/product at these large funds, I’d go with it.
I wrote late last year about founder-investor fit. And in it, I talk about Harry Hurst‘s check-size-to-helpfulness ratio (CS:H). In this ratio, you’re trying to maximize for helpfulness. Ideally, if the fund writes you a $1M check, they’re adding in $10M+ in additive value. And based on a fund’s strategy (i.e. lead investors vs not, $250K or $5M checks, scout programs or solo capitalist + advisory networks, etc.), it’ll determine how helpful they can be to you at the stage you need them.
If you were to plan out your next 18-24 months, take your top three priorities. And specifically, find investors that can help you address those. For example, if you’re looking for intros to potential companies in your sales pipeline and all a VC has to do is send a warm intro to their network/portfolio for you, bigger funds might be more useful. On the other hand, if you’re struggling to find a revenue model for your business, and you need more help than one-offs and quarterly board meetings, I’d look to work with an investor with a smaller portfolio or a solo capitalist. If you’re creating a brand new market, find someone with deep operating experience and domain expertise (even if it’s in an adjacent market), rather than a generalist fund.
While there’s no one-size-fits-all and there are exceptions, here are two ways I think about helpfulness, in other words, value adds:
The uncommon – Differentiators
The common – What everybody else is doing
The uncommon
Of course, this might be the more obvious of the pair. But you’d be surprised at how many founders overlook this when they’re actually fundraising. You want to work with investors that have key differentiators that you need at that stage of your company. By nature of being uncommon, there are million out there. But here are a few examples I’ve seen over the years:
Ability to build communities having built large followings
Content creation + following (i.e. blog, podcast, Clubhouse, etc.)
Getting in’s to top executives at Fortune 500 companies
Closing government contracts
Access/domain expertise on international markets
In-house production teams
They know how to hustle (i.e. Didn’t have a traditional path to VC, yet have some of the biggest and best LPs out there in their fund)
Ability to get you on the front page of NY Times, WSJ, or TechCrunch
Strong network of top executives looking for new opportunities (i.e. EIRs, XIRs)
Influencer network
Category leaders/definers (i.e. Li Jin on the passion economy, Ryan Hoover on communities)
Having all accelerator portfolio founder live under the same roof for the duration of the program (i.e. Wefunder’s XX Fund pre-pandemic)
Surprisingly, not as common as I thought, VCs that pick up your call “after hours”
The common
Packy McCormick, who writes this amazing blog called Not Boring, wrote in one of his pieces, “Here’s the hard thing about easy things: if everyone can do something, there’s no advantage to doing it, but you still have to do it anyway just to keep up.” Although Packy said it in context to founders, I believe the same is true for VCs. Which is probably why we’ve seen this proliferation of VCs claiming to be “founder-friendly” or “founder-first” in the past half decade. While it used to be a differentiator, it no longer is. Other things include:
Money, maybe follow-on investments
Access to the VC’s network (i.e. potential customers, advisors, etc.)
Access to the partner(s) experience
Intros to downstream investors
That said, if an investor is trying to cover all their bases, that is a strategy not to lose rather than a strategy to win, to quote the conversation I had with angel investor Alex Sok recently. As long as it doesn’t come at the expense of their key differentiator. At the same time, it’s important to understand that most VCs will not allocate the same time and energy to every founder in their portfolio. If they are, well, it might be worth reconsidering working with them. It’s great if you’re not a rock-star unicorn. Means you still get the attention and help that you might want. But if you are off to the races and looking to scale and build fast, you won’t get any more help and attention that you’re ‘prescribed’. If you’re winning, you probably want your investor to double down on you.
Even if you’re not, the best investors will still be around to be as helpful as they can, just in more limited spans of time.
Finding investor NPS
You can find CS:H, or investor NPS, out in a couple of ways:
The investors are already adding value to you and your company before investing. Uncommon, but it really gives you a good idea on their value.
You find out by asking portfolio founders during your diligence.
Your founder friends are highly recommending said investor to you.
Then there’s probably the best form of validation. I’ve shared this before, but I still think it’s one of the best indicators of investor NPS. Blake Robbins once quotedBrett deMarrais of Ludlow Ventures, “There is no greater compliment, as a VC, than when a founder you passed on — still sends you deal-flow and introductions.”
In closing
“How likely would you, the founder, recommend this fund/firm/partner(s) to your founder friend(s)?” is a great question to consider when fundraising. But I want to take it a step further. NPS is usually measured on a one to ten scale. But the numbering mechanic is rather nebulous. For instance, an 8/10 on my scale may not equal an 8/10 on your scale. So your net promoter score is more so a guesstimate of the true score. While any surveying question is more or less a guesstimate, I believe this question is more actionable than the above:
If you were to start a new company tomorrow, would you still want this investor on your cap table?
With three options:
No
Yes
It’s a no-brainer.
And if you get two or more “no-brainers”, particularly from (ex-)portfolio startups that fizzled off into obscurity, I’d be pretty excited to work with that investor.
Stay up to date with the weekly cup of cognitive adventures inside venture capital and startups, as well as cataloging the history of tomorrow through the bookmarks of yesterday!
As a venture scout and as someone who loves helping pre-seed/seed startups before they get to the A, I get asked this one question more often than I expect. “David, do you think this is a good idea?” Most of the time, admittedly, I don’t know. Why? I’m not the core user. I wouldn’t count myself as an early adopter who could become a power user, outside of pure curiosity. I’m not their customer. To quote Michael Seibel of Y Combinator,
… “customers are the gatekeepers of the startups world.” Then comes the question, if customers are the gatekeepers to the venture world, how do you know if you’re on to something if you’re any one of the below:
Pre-product,
Pre-traction,
And/or pre-revenue?
This blog post isn’t designed to be the crystal ball to all your problems. I have to disappoint. I’m a Muggle without the power of Divination. But instead, let me share 3 mental models that might help a budding founder find idea-market fit. Let’s call it a tracker’s kit that may increase your chances at finding a unicorn.
Many founders I meet focus on, and rightly so, optimizing their core metrics – a set of units that surprisingly don’t change after its initial inception. But metrics and the way you measure them should undergo constant iteration. Metrics are a way to measure and test your assumptions. 9/10 assumptions, if not all, are honed through the process of iteration. And by transitive property, the metrics we measure, but more importantly, the way we measure them, is subject to no less.
Though I’m not as heavily involved on the operating side as I used to be, although I try to, the bug that inspires me to build never left. So, let’s take it from the perspective of a project a couple friends and I have been working on – hosting events that stretch people’s parameters of ‘possible’. Given our mission, everything we do is to help actuate that. One such metric that admittedly had 2 degrees of freedom from our mission was our NPS score.
The “NPS”
“How likely would you recommend a friend to come to the last event you joined us in?” Measured on a 1-10 scale, we ended up seeing a vast majority, unsurprisingly in hindsight, pick 7 (>85%). A few 9’s, and a negligible amount of 5s, 6s, and 8s. 7 acted as the happy medium for our attendees, all friends, to tell us: “We don’t know how we feel about your event, but we don’t want to offend you as friends.”
We then made a slight tweak, hoping to push them to take a more binary stance. The question stayed the same, but this time, we didn’t allow them to pick 7. In forcing them to pick 8 (a little better than average) and 6 (a little worse than average), we ended up finding all the answers shift to 6s and 8s and nothing else. Even the ones that previously picked 9s regressed to 8s. And the ones who picked 5s picked 6s. Effectively, we created a yes/no question with just this small tweak.
There’s 3 fallacies with this:
Numbers are arbitrary. An 8 for you, may not be an 8 for me. Unless we create a consolidated rubric that everyone follows when answering this question, we’re always going to variability in semi-random expectations.
It’s a lagging indicator. There’s no predictive value in measuring this. By the time they answer this question, they’d already have made their decision. Though the post-mortem is useful, the feedback cycle between events was too long. So, we had to start looking into iterating the event live, or while it was happening.
Answers weren’t completely honest. All the attendees were our friends. So their answers are in part, a reflection of the event, but also in part, to help us ‘save face’.
In studying essentialism, Stoicism, and Rahul Vohra‘s Superhuman, we found a solution that draws on the emotional spectrum that answered 1 and 3 rather well. Instead of phrasing our questions as “How much do you value this opportunity?”, we instead phrased them as “How much would you sacrifice to obtain this opportunity?” Humans are innately loss-averse. Losing your iPhone will affect you more negatively and for longer, than if you won a $1000 lottery.
So, our question transformed into: “How distraught would you be if we no longer invited you to a future event?”, paired with the answers “Very”, “Somewhat”, and “Not at all”. Although I’m shy to say we got completely honest answers, the answers in which we did give allowed for them to follow-up and supplement why they felt that way, without us prompting them.
The only ‘unaddressed’ fallacy by this question – point #2 – was resolved by putting other methods in place to measure attention spans during the event, like the number of times people checked their phone per half hour or the number of unique people who were left alone for longer than a minute per half hour (excluding bio breaks).
Feedback
“How can we improve our event?” We received mostly logistical answers. Most of which we had already noticed either during the event or in our own post-mortem.
In rephrasing to, “How can we help you fall in love with our events?”, we helped our attendees focus on 2 things: (1) more creative responses and (2) deep frustrations that ‘singlehandedly’ broke their experience at the event.
And to prioritize the different facets of feedback, we based it off the answers from the questions:
“What was your favorite element of the event?”
And, “How distraught would you be if we no longer invited you to a future event?”
For the attendees who were excited about elements closely aligned with our mission, we put them higher on the list. There were many attendees who enjoyed our event for the food or the venue, though pertinent to the event’s success, fell short of our ultimate mission. That said, once in a while, there’s gold in the feedback from the latter cohort.
On the flip side, it may seem intuitive to prioritize the feedback of those who were “Very distraught” or “Not at all”. But they exist on two extremes of the spectrum. One, stalwart champions of our events. The other, emotionally detached from the success of our events. In my opinion, neither cohort see our product truly for both its pros and cons, but rather over-index on either the pros or the cons, respectively. On a slight tangent, this is very similar to how I prioritize which restaurants to go to or which books to read. So, we find ourselves prioritizing the feedback of the group that lie on the tipping point before they “fall in love” with our events.
Unscalability and Scalability
We did all of our feedback sessions in-person. No Survey Monkey. No Google Forms, Qualtrics, or Typeform. Why?
We could react to nuances in their answers, ask follow-up questions, and dig deeper.
We wanted to make sure our attendees felt that their feedback was valued, inspired by Google’s Project Aristotle.
And, in order to get a 100% response rate.
We got exactly what we expected. After our post-mortem, as well as during the preparation for our next event, I would DM/call/catch up with our previous attendees and tell them which feedback we used and how much we appreciated them helping us grow. For the feedback we didn’t use, I would break down what our rationale was for opting for a different direction, but at the same time, how their feedback helped evolve the discourse around our strategic direction. Though their advice was on the back burner now, I’ll be the first to let them know when we implement some element of it.
The flip side of this is that it looks extremely unscalable. You’re half-right. Our goal isn’t to scale now, as we’re still searching for product-market fit. But as you might notice, there are elements of this strategy that can scale really well.
In closing
Of course, our whole endeavor is on hold during this social distancing time, but the excitement in finding new and better ways to measure my assumptions never ceases. So, in the interim, I’ve personally carried some of these interactions online, in hopes of discovering something about virtual conversations.
