Welcome to the RSCM Blog!

As regular readers know, I'm a huge fan of the AIPP data set collected by Rob Wiltbank.  In this TechCrunch article from a while back, Rob corrected the mistaken assertion that angels don't make money.  He also said that, "Angel investors probably should look to make at least a dozen investments..." While this statement isn't wrong, neither is it as nuanced as I would like.‍

In this previous post, I used simulated data to examine the issue of adequate diversification. As it happens, I've also privately analyzed this issue using Rob's AIPP data, an analysis I will now make public. [Hat tip to Paul P, a colleague with a professional financial statistics background who inspired and reviewed the original analysis, though any mistakes remain mine.]

To achieve my fully nuanced answer, I need to address two issues. But I'll devote separate posts to each issue.  This post addresses the issue that diversification is a tradeoff.  More diversification will always protect you slightly more from idiosyncratic risk, all other things being equal (see my diversification posts one and two). But diversification isn't free. It's harder to make 10 investments than 1, and harder still to make 100. As an investor, you want to know how much each additional investment reduces idiosyncratic risk so you can make the right tradeoff for you.

To demonstrate this tradeoff, I'll focus on RSCM's particular investing strategy: seed stage, no follow on, technology sector.  I documented how I applied this strategy to the AIPP data set, complete with filtered data file, in this post. In principle, you could create a filtered data file for your own strategy as described in this other post and then apply the procedure below to calculate the precise diversification tradeoff for your strategy.

The key to the whole problem is a statistical technique called resampling; I'll take the records from my filtered data file and create 10K portfolios for a variety of portfolio sizes, using random sampling with replacement. Then it's straightforward to determine how much the historical returns at each portfolio size could vary. Essentially, we're performing the following thought experiment: what if 10K angels randomly invested in the startups that Rob studied?  What if they each invested in 10, 20, 30,... companies?  How likely would a given angel have been to achieve a given return?

The return for the entire sample is a 42% IRR and a 3.7x payout multiple.  The graph below shows the probability of achieving at least 1x, 2x, and 3x the original investment for a range of portfolio sizes. [Excel file with data, sampling macro, and probability macro is here.]

Now you can see the diversification tradeoff.  Personally, I think a bare minimum is a 90% chance of getting at least your money back. That's 15 investments based on the historical data. Roughly the same as Rob's answer of a "dozen". But I'd really prefer a 90% chance of doubling my money. That's 70 investments. Now, if I'm managing other people's money, I'd really like to push to an 80% chance of tripling their money, which is over 200 investments. I didn't run 400 investments, but I'm guessing that is roughly the point at which you would have had a 90% chance of tripling other people's money. [Update 4/2/2013: after doing a bunch or runs, it looks like it takes over 500 investments to achieve a 90% chance of tripling.] As far as I know, 500 Startups is the only firm other than RSCM that is even shooting for that level of diversification in a single portfolio.So there's my take on the first issue, diversification as a tradeoff. The second issue is essentially the old problem of, "Past performance is no guarantee of future results." What if the return distribution of seed stage technology startups is different going forward than during the period Rob collected data? It turns out there's a nifty way to use resampling to test the sensitivity of different levels of diversification to shifts in the return curve. That's the topic of the next post.[Updated 4/25/2013: corrected minor error in spreadsheet and graph]

One of the questions I most frequently answer about RSCM is how we value seed stage startups. Apparently, being not only willing, but eager to set equity valuations sets us apart from the vast majority of investors. It's also the aspect of our approach that I'm most proud of intellectually. Developing the rest of our process was mostly a matter of basic data analysis and applying existing research. But the core of our valuation system rests on a real (though modest) insight.

We've finally accumulated enough real-world experience with our valuation approach that I feel comfortable publicly discussing it. Now, I'm not going to give out the formula. Partly, this is to preserve some semblance of unique competitive advantage. But it's also for practical reasons:

• Our precise formula is tuned for our specific investment theses, which are based on our larger analysis of exit markets, technology dynamics, and diversification requirements.
• The current version of the formula doesn't communicate just how adaptable the fundamental concept is (and we do in fact adjust it as we learn).
• There's a lot of truth in the wisdom about teaching a man to fish rather than giving him a fish.

Instead, I'm going to discuss how we constructed the formula. Then you can borrow whatever aspects of our approach you think are valid (if any) and build your own version if you like.The first part of our modest insight was to face the fact that, at the seed stage, most of the value is option value not enterprise value. Any approach based on trying to work backwards from some hypothetical future enterprise value will be either incredibly expensive or little more than a guess. But how do you measure a startup's option value from a practical standpoint?The second part of our modest insight was to ask, "Is there anyone who has a big stake in accurately comparing the unknown option value to some other known dollar value?"  The answer was obvious once we formulated the question: the founders. If the option value of their ownership stake were dramatically less, on a risk-adjusted basis, than what they could earn working for someone else, they probably wouldn't be doing the startup. Essentially, we used the old economist's trick of "revealed preference".We knew there could be all sorts of confounding factors. But there might be a robust relationship between founders' fair market salaries and their valuation. So we tested the hypothesis. We looked at a bunch of then current seed-stage equity deals where we knew people on the founder or investor side, or the valuation was otherwise available. We then reviewed the founders' LinkedIn profiles or bios to estimate their salaries.What we found is that equity valuations for our chosen segment of the market tended to range from 2x to 4x the aggregate annual salary of the founders. The outliers seemed to be ones that either (a) had an unusual amount of "traction", (b) came out of a premier incubator, or (c) were located in the Bay Area. Once we controlled for these factors, the 2x to 4x multiple was even more consistent.Now, the concept of a valuation multiple is pretty common. In the public markets, equity analysts and fund managers often use the price-to-earnings ratio. For later stage startups, venture capitalists and investment bankers often use the revenue multiple. Using a multiple as a rule-of-thumb allows people to:

• Compare different sectors, e.g., the P/E ratios in technology are higher than in retail.
• Compare specific companies to a benchmark, e.g., company X appears undervalued.
• Set valuations, e.g., for IPOs or acquisitions.

Obviously, 2x to 4x is a big range. The next step was to figure out what drives the variance. Here, we relied on the research nicely summarized in Sections 3.2-3.6 of Hastie and Dawes' Rational Choice in an Uncertain World. In high-complexity, high-uncertainty environments, experts are pretty bad at making overall judgements. But they are pretty good at identifying the key variables. So if all you do is poll experts on the important variables and create a consensus checklist, you will actually outperform the experts. The explanation for this apparent paradox is that the human brain has trouble consistently combining multiple factors and ignoring irrelevant information (such as whether the investor personally likes the founders) when making abstract judgements.

So that's what we did. We asked highly experienced angels and VCs what founder characteristics are most important at the seed stage. (We focused on the founders because we had already determined that predicting the success of ideas this early was hopeless.) The most commonly mentioned  factors fell into the general categories you'd expect: entrepreneurial experience, management experience, and technical expertise. Going to a good undergraduate or graduate program were also somewhat important. Our experts further pointed out that making initial progress on the product or the business was partly a reflection on the founders' competence as well as the viability of the idea.We created a checklist of points in these categories and simply scaled the valuation multiple from 2x to 4x based on the number of points. Then we tested our formula against deals that were actually in progress, predicting the valuation and comparing this prediction to the actual offer. This initial version performed pretty well. We made some enhancements to take into account location, incubator attendance, and the enterprise value of progress, then tested again. This updated version performed very well. Finally, we used our formula to actually make our own investments. The acceptance rate from founders was high and other investors seemed to think we got good deals.Is our formula perfect?  Far from it. Is it even good? Truthfully, I don't know. I don't even know what "good" would mean in the abstract. Our formula certainly seems far more consistent and much faster than what other investors do at the seed stage. Moreover, it allows us to quickly evaluate deal flow sources to identify opportunities for systematically investing in reasonably valued startups. These characteristics certainly make it very useful.

I'm pretty confident other investors could use the same general process to develop their own formulas, applicable to the particular categories of startups they focus on—as long as these categories are ones where the startups haven't achieved a clear product-market fit. Past that point, enterprise value becomes much more relevant and amenable to analysis, so I'm not sure the price-to-salary multiple would be as useful.

I privately received a couple of interesting comments on my diversification post:

One of RSCM's angel advisors wrote, "I would think most smart people get it intellectually, but many are stuck in the mindset that they have a particular talent to pick winners."

One of my Facebook friends commented, "VC seems to be a game of getting a reputation as a professional die thrower."

I pretty much agree with both of these statements. However, even if you believe someone has mad skillz at die-rolling, you may still be better off backing an unskilled roller. Diversification is that powerful! To illustrate, consider another question:

Suppose I offered you a choice between the following two options:

(a) You give me $1M today and I give you somewhere between $3M and $3.67M with 99.99% certainty in 4 years.

(b) You give me $1M today and a "professional" rolls a standard six-sided die.  If it comes up a 6, I give you $20M in 4 years. Otherwise, you lose the $1M. But this guy is so good, he never rolls a 1 or 2.

The professional's chance of rolling a 6 is 25% because of his skill at avoiding 1s and 2s. So option (b) has an expected value of $5M. Option (a) only has an expected value of $3.33M. Therefore, the professional has a 50% edge. But he still has a 75% chance of losing all your money.I'm pretty sure that if half their wealth were on the line, even the richest players would chose (a).  Those of you who read the original post probably realize that option (a) is actually an unskilled roller making 10,000 rolls.  Therefore:

Diversifying across unskilled rolls can be more attractive than betting once on a skilled roller.

Of course, 1 roll versus 10,000 hardly seems fair.  I just wanted to establish the fact that diversification can be more attractive than skill in principle.  Now we can move on to understanding the tradeoff.To visualize diversification versus skill, I've prepared two graphs (using an enhanced version of my diversification spreadsheet).  Each graph presents three scenarios: (1) an unskilled roller with a standard 1 in 6 chance of rolling a 6, (2) a somewhat skilled roller who can avoid 1s so has a 1 in 5 chance of rolling a 6, and (3) our very skilled roller who can avoid 1s and 2s so has a 1 in 4 chance of rolling a 6.First, let's look at how the chance of at least getting your money back varies by the number of rolls and the skill of the roller:

The way to interpret this chart is to focus on one of the horizontal gray lines representing a particular probability of winning your money back and see how fast the three curves shift right.  So at the 0.9 "confidence level", the very skilled roller has to make 8 rolls, the somewhat skilled roller has to make 11, and the unskilled roller has to make 13.

From the perspective of getting your money back, being very skilled "saves" you about 5 rolls at the 0.9 confidence level. Furthermore, I'm quite confident that most people would strongly prefer a 97% chance of at least getting their money back with an unskilled roller making 20 rolls to the 44% chance of getting their money back with a very skilled roller making 2 rolls, even though their expected value is higher with the skilled roller.Now let's look at the chance of winning 2.5X your money:

The sawtooth pattern stems from the fact that each win provides a 20X quantum of payoff.  So as the number of rolls increases, it periodically reaches a threshold where you need one more win, which drops the probability down suddenly.Let's look at the 0.8 confidence level.  The somewhat skilled roller has a 2 to 5 roll advantage over the unskilled roller, depending on which sawtooth we pick.  The very skilled roller has a 3 roll advantage over the unskilled roller initially, then completely dominates after 12 rolls. Similarly, the very skilled roller has a 2 to 5 roll advantage over the somewhat skilled roller, dominating after about 30 rolls.

Even here, I think a lot of people would prefer the 76% chance of achieving a 2.5X return resulting from the unskilled roller making 30 rolls to the 58% chance resulting from the very skilled roller making 3 rolls.But how does this toy model generalize to startup investing? Here's my scorecard comparison:

• Number of Investments. When Rob Wiltbank gathered the AIPP data set on angel investing, he reported that 121 angel investors made 1,038 investments. So the mean number of investments in an angel's portfolio was between 8 and 9. This sample is probably skewed high due to the fact that it was mostly from angels in groups, who tend to be more active (at least before the advent of tools like AngelList).  Therefore, looking at 1 to 30 trials seems about right.

• "Win" Probability. When I analyzed the subset of AIPP investments that appeared to be seed-stage, capital-efficient technology companies (a sample I generated using the methodology described in this post), I found that the top 5% of outcomes accounted for 57% of the payout. That's substantially more skewed than a 1 in 6 chance of winning 20X.  My public analysis of simulated angel investment and an internal resampling analysis of AIPP investments bear this out. You want 100s of investments to achieve reasonable confidence levels. Therefore, our toy model probably underestimates the power of diversification in this context.

• Degree of Skill. Now, you may think that there are so many inexperienced angels out there that someone could get a 50% edge. But remember that the angels who do well are the ones that will keep investing and angels who make lots of investments will be more organized. So there will be a selection effect towards experienced angels. Also, remember that we're talking about the seed stage where the uncertainty is the highest. I've written before about how it's unlikely one could have much skill here. If you don't believe me, just read chapters 21 and 22 of Kahneman's Thinking Fast and Slow. Seed stage investment is precisely the kind of environment where expert judgement does poorly. At best, I could believe a 20% edge, which corresponds to our somewhat skilled roller.

The conclusion I think you should draw is that even if you think you or someone you know has some skill in picking seed stage technology investments, you're probably still better at focusing on diversification first.  Then try to figure out how to scale up the application of skill.

And be warned, just because someone has a bunch of successful angel investments, don't be too sure he has the magic touch. According to the Center for Venture Research, there were 318,000 active angels in the US last year. If that many people rolled a die 10 times, you'd expect over 2,000 to achieve at least a 50% hit rate purely due to chance! And you can bet that those will be the people you hear about, not the 50,000 with a 0% hit rate, also purely due to chance.

In science, there isn't really any such thing as a "fact".  Just different degrees of how strongly the evidence supports a theory. But diversification is about as close as we get. Closer even than evolution or gravity. In "fact", neither evolution or gravity would work if diversification didn't.So I've been puzzled at some people's reaction to RSCM's startup investing strategy.  They don't seem to truly believe in diversification. I can't tell if they believe it intellectually but not emotionally or rather they think there is some substantial uncertainty about whether it works.In either case, here's my attempt  at making the truth of diversification viscerally clear.  It starts with a question:

Suppose I offered you a choice between the following two options:

(a) You give me $1M today and I give you $3M with certainty in 4 years.

(b) You give me $1M today and we roll a standard six-sided die.  If it comes up a 6, I give you $20M in 4 years. Otherwise, you lose the $1M.

Option (b) has a slightly higher expected value of $3.33M, but an 83.33% chance of total loss. Given the literature on risk preference and loss aversion (again, I highly recommend Kahneman's book as an introduction), I'm quite sure the vast majority of people will chose (a).  There may be some individuals, enterprises, or funds who are wealthy enough that a $1M loss doesn't bother them.  In those cases, I would restate the offer.  Instead of $1M, use $X where $X = 50% of total wealth. Faced with an 83.33% chance of losing 50% of their wealth, even the richest player will almost certainly chose (a).Moreover, if I took (a) off the table and offered (b) or nothing, I'm reasonably certain that almost everyone would choose nothing. There just aren't very many people willing to risk a substantial chance of losing half their wealth. On the other hand, if I walked up to people and credibly guaranteed I'd triple their money in 4 years, almost everyone with any spare wealth would jump at the deal.

Through diversification, you can turn option (b) into option (a).

This "trick" doesn't require fancy math.  I've seen people object to diversification because it relies on Modern Portfolio Theory or assumes rational actors.  Not true.  There is no fancy math and no questionable assumptions. In fact, any high school algebra student with a working knowledge of Excel can easily  demonstrate the results.Avoiding Total LossLet's start with the goal of avoiding a total loss. As Kahneman and Tversky showed, people really don't like the prospect of losing large amounts. If you roll the die once, your chance of total loss is (5/6) = .83.  If you roll it twice, it's (5/6)^2 = .69.  Roll it ten times, it's (5/6)^10 = .16. The following graph shows how the chance of total loss rapidly approaches zero as the number of rolls increases.

By the time you get to 50 rolls, the chance of total loss is about 1 in 10,000.  By 100 rolls, it's about 1 in 100,000,000.  For comparison, the chance of being struck by lightning during those same four years is approximately 1 in 200,000 (based on the NOAA's estimate of an annual probability of 1 in 775,000).

‍Tripling Your Money

‍Avoiding a total loss is a great step, but our ultimate question is how close can you get to a guaranteed tripling of your money.  Luckily, there's an easy way to calculate the probability of getting at least a certain number of 6s using the Binomial Theorem (which has been understood for hundreds of years).  One of many online calculator's is here. I used the BINOMDIST function of Excel in my spreadsheet.

The next graph shows the probability of getting back at least 3x your money for different numbers of rolls.  The horizontal axis is logarithmic, with each tick representing 1/4 of a power of 10.

As you can see,  diversification can make tripling your money a near certainty. At 1,000 rolls, your probability of at least tripling up is 93%. And with that many rolls, Excel can't even calculate the probability of getting back less than your original investment. It's too small. At 10,000 rolls, the probability of less than tripling your money is 1 in 365,000.So if you have the opportunity to make legitimate high-risk, high-return investments, your first question should be how to diversify. All other concerns are very secondary.

Now, I will admit that this explanation is not the last word. Our model assumes independent, identical bets with zero transaction costs. If I have time and there's interest, I'll address these issues in future posts. But I'm not sweeping them under the rug. I'm truly not aware of any argument that their practical effect would be significant with regards to startup investments.

What do you do when you have to make decisions in an uncertain environment with only mediocre data?  Startup founders and investors face this question all the time.I had an interesting email exchange on this topic with Brad Feld of Foundry Group. First, let me say that I like Brad and his firm.  If I were the founder of a startup for whom VC funding made sense, Foundry would be on my short list.

Now, Brad has an Master's in Management Science from MIT and was in the PhD program. I have a Master's in Engineering-Economic Systems from Stanford, specializing in Decision Theory.  So we both have substantial formal training in analyzing data and are both focused on investing in startups.

But we evidently take opposing sides on the question of how data should inform decision-making. Here's a highly condensed version of our recent conversation on my latest "Seed Bubble" post (don't worry, I got Brad's permission to excerpt):

Brad: Do you have a detailed spreadsheet of the angel seed data or are you using aggregated data for this?... I'd be worried if you are basing your analysis... without cleaning the underlying data.

Kevin:  It's aggregated angel data....   I'm generally skeptical of the quality of data collection in both... data sets.... But the only thing worse than using mediocre data is using no data.

Brad: I hope you don't believe that. Seriously - if the data has selection bias or survivor bias, which this data likely does, any conclusions you draw from it will be invalid.

Kevin: ...of course I believe it....  Obviously, you have to assess and take into account the data's limitations... But there's always some chance of learning something from a non-empty data set.  There's precisely zero chance of learning something from nothing.

Brad: ... As a result, I always apply a qualitative lens to any data (e.g. "does this fit my experience"), which I know breaks the heart of anyone who is purely quantitative (e.g. "humans make mistakes, they let emotions cloud their analysis and judgement").

I don't want to focus on these particular data sets.  Suffice it to say that I've thought reasonably carefully about their usefulness in the context of diagnosing a seed investment bubble.  If anyone is really curious, let me know in the comments.Rather, I want to focus on Brad's and my positions in general. I absolutely understand Brad's concerns.  Heck, I'm a huge fan of the "sanity check".  And I, like most people with formal data analysis training, suffer a bit from How The Sausage Is Made Syndrome.  We've seen the compromises made in practice and know there's some truth to Mark Twain's old saw about "lies, damned lies, and statistics." When data is collected by an industry group rather than an academic group (as is the case with the NVCA data) or an academic group doesn't disclose the details of their methodology (as is the case with the CVR angel data), it just feeds our suspicions.I think Brad zeroes in on our key difference in the last sentence quoted above:

...which I know breaks the heart of anyone who is purely quantitative (e.g. "humans make mistakes, they let emotions cloud their analysis and judgement").

I'm guessing that Brad thinks the quality of human judgement is mostly a matter of opinion or that it can be dramatically improved with talent/practice.  Actually, the general inability of humans to form accurate judgements in uncertain situations has been thoroughly established and highly refined by a large number of rigorous scientific studies, dating back to the 1950s.  It's not quite as "proven" as gravity or evolution, but it's getting there.At Stanford, I mostly had to read the original papers on this topic.  Many of them are, shall we say, "difficult to digest." But now, there are several very accessible treatments.  For a general audience, I recommend Daniel Kahneman's Thinking Fast and Slow, where he recounts his journey exploring this area, from young researcher to Nobel Prize winner.  For a more academic approach, I recommend Hastie's and Dawes' Rational Choice In an Uncertain World. If you need to make decisions in uncertain environments and aren't already familiar with the literature, I cannot recommend strongly enough reading at least one of these books.

But in the meantime, I will sum up.  Human's are awful at forming accurate judgements in situations where there's a lot of uncertainty and diversity (known as low validity environments).  It doesn't matter if you're incredibly smart.  It doesn't matter if you're highly experienced.  It doesn't even matter if you know a lot about cognitive biases.  The fast, intuitive mechanisms your brain uses to reach conclusions just don't work well in these situations. If the way quantitative data analysis works in practice gives you pause, the way your brain intuitively processes data should have you screaming in horror.

Even the most primitive and ad hoc quantitative methods  (such as checklists) generally outperform expert judgements, precisely because they disengage the intuitive judgment mechanisms. So if you actually have a systematically collected data set, even if you think it almost certainly has some issues, I say the smart money still heavily favors the data rather than the expert.

By the way, lots of studies also show that people tend to be overconfident. So thinking that you have a special ability or enough expertise so that this evidence doesn't apply to you... is probably a cognitive illusion too. I say this as a naturally confident guy who constantly struggles to listen to the evidence rather than my gut.

My recommendation: if you're in the startup world, by all means, have the confidence to believe you will eventually overcome all obstacles. But when you have to make an important estimate or a decision, please, please, please, sit down and calculate using whatever data is available.  Even if it's just making a checklist of your own beliefs.

Earlier this year, I showed that there was little hard evidence of a general bubble in seed-stage investing. As this recent TechCrunch article shows, the meme has persisted. So I thought I'd take another look to see if anything has changed.I re-crunched the CVR and NVCA data, including the new information for 1H2011 (which I annualized to make the numbers comparable). Bottom line: there has been a slight recovery in the angel contribution and continued growth in the superangel segment. But these increases have been mostly offset by a decrease inVC seed activity. (My collation of the data is available in this Excel file.) Here are updated version of the dollar volume charts.

This is about what I expected. I think angels' willingness to invest is driven primarily by the macro environment, which has been improving, albeit rather slowly. I think LPs willingness to give VCs more dollars to invest is driven by both the macro environment and historical fund returns, which have been very poor.Now I was a little surprised at the super angel situation. I had expected a really dramatic expansion from super angels. First, I searched for new super angels using TechCrunch, VentureBeat, and Google. I only found two. IMAF (focused on North Carolina) and Michael Arrington's CrunchFund (no Web site as of this posting). According to their SEC Form Ds, they are $13M and $16M respectively.Second, I searched the SEC Edgar database for all the funds on the original list from Chubby Brain. Other than Quest Venture Partners, I was able to locate filings for all the significant funds. Jeff Clavier's SoftTech VC and Ron Conway's SV Angel both had decent increases, from $15M to $35M and $20M to $40M respectively. But in my opinion, those two have reputations such that they could support much larger funds. Equally strong were Lerer Ventures' increase from $7M to $25M and Thrive Capital's increase from $10M to $40M.The big winner was Roger Ehrenberg 's IA Ventures with a jump from $25M to $100M!But nobody else has appeared to raise a new fund. Even with these increases, the total confirmed super angel dollars "only" rose from $253M to $440M. That's a lot, but not the $1B I would have guessed given the press coverage. Also, a ~$200M boost spread over multiple years just isn't that significant when you're talking about a market that is $8.5B per year.So I'll stick to my guns. No general seed bubble (at least for now).

Several people have pointed me to Dan Frommer’s post on Moneyball for Tech Startups, noting that “Moneyball” is actually a pretty good summary of our approach to seed-stage investing at RSCM. Steve Bennet, one of our advisors and investors, went so far as to kindly make this point publicly on his blog.Regular readers already know that I’ve done a fair bit of Moneyball-type analysis using the available evidence for technology startups (see here, here, here, here, here, and here). But I thought I’d take this opportunity to make the analogy explicit.I’d like to start by pointing out two specific elements of Moneyball, one that relates directly to technology startups and one that relates only indirectly:

• Don’t trust your gut feel, directly related. There’s a quote in the movie where Beane says, “Your gut makes mistakes and makes them all the time.” This is as true of tech startups as it is of baseball prospects. In fact, there’s been a lot of research on gut feel (known in academic circles as “expert clinical judgement”). I gave a fairly detailed account of the research in this post, but here’s the summary. Expert judgement never beats a statistical model built on a substantial data set. It rarely even beats a simple checklist, and then only in cases where the expert sees thousands of examples and gets feedback on most of the outcomes. Even when it comes to evaluating people, gut feel just doesn’t work. Unstructured interviews are the worst predictor of job performance.
• Use a “player” rating algorithm, indirectly related. In Moneyball, Beane advocates basing personnel decisions on statistical analyses of player performance. Of course, the typical baseball player has hundreds to thousands of plate appearances, each recorded in minute detail. A typical tech startup founder has 0-3 plate appearances, recorded at only the highest level. Moreover, with startups, the top 10% of the startups account for about 80% of the all the returns. I’m not a baseball stats guy, but I highly doubt the top 10% of players account for 80% of the offense in the Major Leagues. So you’ve got much less data and much more variance with startups. Any “player” rating system will therefore be much worse.

Despite the difficulty of constructing a founder rating algorithm, we can follow the general prescription of trying to find bargains. Don’t invest in “pedigreed” founders, with startups in hot sectors, that have lots of “social proof”, located in the Bay Area. Everyone wants to invest in those companies. So, as we saw in Angel Gate, valuations in these deals go way up. Instead, invest in a wide range of founders, in a wide range of sectors, before their startups have much social proof, across the entire US. Undoubtedly, these startups have a lower chance of succeeding. But the difference is more than made up for by lower valuations. Therefore, achieving better returns is simply a matter of adequate diversification, as I’ve demonstrated before.Now, to balance out the disadvantage in rating “players”, startup investors have an advantage over baseball managers. The average return of pure seed stage angel deals is already plenty high, perhaps over 40% IRR in the US according to my calculation. You don’t need to beat the market. In fact, contrary to popular belief, you don’t even need to try and predict “homerun” startups. I’ve shown you’d still crush top quartile VC returns even if you don’t get anything approaching a homerun. Systematic base hits win the game.But how do you pick seed stage startups? Well, the good news from the research on gut feel is that experts are actually pretty good at identifying important variables and predicting whether they positively or negatively affect the outcome. They just suck at combining lots of variables into an overall judgement. So we went out and talked to angels and VCs. Then, based on the the most commonly cited desirable characteristics, we built a simple checklist model for how to value seed-stage startups.We’ve made the software that implements our model publicly available so anybody can try it out [Edit 3/16/2013: we took down the Web app in Jan 2013 because it wasn’t getting enough hits anymore to justify maintaining it. We continue to use the algorithm internally as a spreadsheet app]. We’ve calibrated it against a modest number of deals. I’ll be the first to admit that this model is currently fairly crude. But the great thing about an explicit model is that you can systematically measure results and refine it over time. The even better thing about an explicit model is you can automate it, so you can construct a big enough portfolio.That’s how we’re doing Moneyball for tech startups.

In spreading the word about RSCM, I recently encountered a question that led to some interesting findings.  A VC from a respected firm, known for its innovative approach, brought up the issue of "homeruns".  In his experience, every successful fund had at least one monster exit.  He was concerned that RSCM would never get into those deals and therefore, have trouble generating good returns.

My initial response was that we'll get into those deals before they are monsters.  We don't need the reputation of a name firm because the guys we want to fund don't have any of the proof points name firms look for.  They'll attract the big firms some time after they take our money.  Of course, this answer is open to debate.  Maybe there is some magical personal characteristics that allows the founders of Google, Facebook, and Groupon to get top-tier interest before having proof points.So I went and looked at the data to answer the question, "What if we don't get any homeruns at all?"  The answer was surprising.I started with our formal backtest, which I produced using the general procedure described in a previous post.  It used the criteria of no follow-on and stage <= 2, as well as eliminating any company in a non-technology sector or capital-intensive one such as manufacturing and biotechnology.

Now, the AIPP data does not provide the valuation of the company at exit.  However, I figured that I could apply increasingly stringent criteria to weed out any homeruns:

1. The payout to the investor was < $5M.
2. The payout to the investor was < $2.5M
3. The payout to the investor was < $2.5M AND the payout multiple was < 25X.

‍

It's hard to imagine an investment in any big winner that wouldn't hit at least the third threshold.  In fact, even scenarios (1) and (2) are actually pretty unfair to us because they exclude outcomes where we invest $100K for 20% of a startup, get diluted to 5-10%, and then the company has a modest $50M exit.  That's actually our target investment!  But I wanted to be as conservative as possible.The base case was 42% IRR and a 3.7x payout multiple.  The results for the three scenarios are:

1. 42% IRR, 2.7x multiple
2. 36% IRR, 2.4x multiple
3. 29% IRR, 2.1x multiple

‍

Holy crap!  Even if you exclude anything that could be remotely considered a homerun, you'd still get a 29% IRR!As you can see, the multiple goes down more quickly than the IRR. Large exits take longer than small exits so when you exclude the large exits, you get lower hold times, which helps maintain IRR.  But that also means you could turn around and reinvest your profits earlier.  So IRR is what you care about from an asset class perspective.For comparison, the top-quartile VC funds currently have 10-year returns of less than 10% IRR, according to Cambridge Associates.  So investing in an index of non-homerun startups is better than investing in the funds that are the best at picking homeruns. (Of course, VC returns could pick up if you believe that the IPO and large acquisition market is going to finally make a comeback after 10 years.)

I've got to admit that the clarity of these results surprised even me.  So in the words of Adam Savage and Jamie Hyneman, "I think we've got to call this myth BUSTED."(Excel files: basecase, scenario 1, scenario 2, scenario 3)

At the moment, people seem to believe there's a "bubble" in seed-stage technology funding.  Many limited partner investors in VC funds I've spoken with have raised the concern and related topics seem popular on Quora (see here, here, and here).  However, I've examined the data and it argues pretty strongly against a widespread seed-stage bubble.Rather, I think the increased attention that top startups attract these days induces availability bias.  Because Y Combinator and superangels generate pretty intense media coverage, people read more frequently about the few big investments in seed-stage startups.  They confuse the true frequency of high valuations with the amount of coverage.  Of course, they never read about all the other seed-stage startups that don't get high valuations.But if you look at the data on the aggregate amount of seed funding and the average deal size, I think it's very hard to argue for a general seed-stage bubble.  At worst, there may be a very localized bubble centered around consumer Internet startups based in the Bay Area.

First, look at the amount of seed funding by angels over the last nine years, as reported by the Center for Venture Research.  I calculated the amount for each year by multiplying the reported total amount of funding by the reported percentage going to seed and early stage deals.  (Note: for some reason the CVR didn't report the percentage in 2004, so I interpolated that data).

As you can see, the amount of seed funding by angels in 2009-20010 was down by half from its level in 2004-2006.  Hard to have a bubble when you're only investing 50% of the dollars you were at the recent peak.  But perhaps it's a pricing issue and angels are pumping more dollars into each startup.  While the CVR doesn't break down the average investment amount at each stage, we can calculate the average investment amount across all stages and use it as a rough index for what is probably going on at the seed and early stage (the index of 100 corresponds to a $436K investment).

The amount invested in each startup in 2010 was down 35% from its 2006 peak.  Now, the investment amount is not the same as the valuation.  However, for a variety of reasons (anchoring on historical ownership, capitalization table management, and price  equilibrium for the marginal startup), I doubt angels have radically changed the percentage of a company they try to own.  So deal size shifts should be a good proxy for valuation shifts.Now, you might think that VC moves in the seed stage market could be a factor.  Probably not, for two reasons.  First, VCs account for a much smaller share of the seed stage market.  Second, what gets counted as the seed stage in the VC data isn't what most of us think of as seed stage investments.  Check out the seed dollar chart and the average seed investment data from the National Venture Capital Association.

Notice that amount of seed funding by VCs has remained flat for the last three years.  Moreover, angels invest dollars in the seed stage at a rate of 3:1 compared to VCs.  So VCs probably aren't contributing to a widespread seed bubble.  But the story takes a strange twist if you look at the average size of VCs' seed stage investments.

The size has increased since 2007.  But look at the absolute level!  $4M+ seed rounds?  I'm starting to think that "seed" does not mean the same thing to VCs as it does to angels and entrepreneurs.  Obviously, VCs cannot be affecting what I think of as the seed round very much.  However, they could be generating the impression of a bubble by enabling a few "mega-seed" deals.  VCs did 373 seed deals in 2010 while angels did around 20,000 (NVCA and CVR data, respectively).

The last factor we have to account for is the superangels.  Most of them are not members of the NVCA.  However, they probably aren't counted by the CVR surveys of individual angels and angel groups either.  ChubbyBrain has a list of the superangels that seems pretty complete; I can't think of anyone I consider a superangel who isn't on it.  Of the 16, there are known fund sizes for 13.  Two of them (Felcis and and SoftTech VC) are members of the NVCA and thus included in that data.  The remaining 11 total $253M.Now, there are probably some smaller, lesser known superangels not on this list.  However, many on the list will not invest all their dollars in a single year and some will invest dollars in follow-on rounds past the seed stage.  So I'm confident that $253M is a generous estimate of the superangel dollars that go into the seed stage each year.  That's only about 3% of angels and VCs combined.Just to really drive the point home, here's a graph of all seed dollars, assuming superangels did $253M per year in 2009 and 2010.  Seed funding is down $5.4B or 40% from it's peak in 2005! So I don't believe there's a bubble.

(The spreadsheet with all my data is here.)

According to our Web statistics, my post on Angel Investing Returns was pretty popular, so I thought I'd dive a little deeper into the process of extracting information from this data set.  At the end of the last post, I hinted that there might be some value in, "...analyzing subsets of the AIPP data..."  Why would you want to do this?  To test hypotheses about angel investing.Now, you must be careful here.  You should always construct your hypotheses before looking at the data.  Otherwise, it's hard to know if this particular data is confirming your hypothesis or if you molded your hypothesis to fit this particular data.  You already have the challenge of assuming that past results will predict future results.  Don't add to this burden by opening yourself to charges of "data mining".I can go ahead and play with this data all I want.  I already used it to "backtest" RSCM's investment strategy.  We developed it by reading research papers, analyzing other data sources, and running investment simulations.  When we found the AIPP download page, it was like Christmas: a chance to test our model against new data.  So I already took my shot.  But if you're thinking about using the AIPP data in a serious way, you might want to stop reading unless you've written your hypotheses down already.  As they say, "Spoiler alert."But if you're just curious, you might find my three example hypothesis tests interesting.  They're all based loosely on questions that arose while doing research for RSCM.

Hypothesis 1: Follow On Investments Don't Improve Returns

It's an article of faith in the angel and VC community that you should "double down on your winners" by making follow on investments in companies that are doing well.  However, basic portfolio and game theory made me skeptical.  If early stage companies are riskier, they should have higher returns.  Investing in later stages just mixes higher returns with lower returns, reducing the average.  Now, some people think they have inside information that allows them to make better follow-on decisions and outperform the later stage average.  Of course, other investors know this too.  So if you follow on in some companies but not others, they will take it as a signal that the others are losers.  I don't think an active angel investor could sustain much of an advantage for long.But let's see what the AIPP data says.  I took the Excel file from my last post and simply blanked out all the records with any follow on investment entries.  The resulting file with 330 records is here.  The IRR was 62%, the payout multiple was 3.2x, and the hold time was 3.4 years.  That's a huge edge over 30% and 2.4x!Now, let's not get too excited here.  There's a difference between deals where there was no follow on and deals where an investor was using a no-follow-on strategy.  We don't know why an AIPP deal didn't have any follow on.  It could be that the company was so successful it didn't need more money.  Of course, the fact that this screen still yields 330 out of 452 records argues somewhat against a very specific sample bias, but there could easily be more subtle issues.Given the magnitude of the difference, I do think we can safely say that the conventional wisdom doesn't hold up.  You don't need to do follow on. However, without data on investor strategies, there's still some room for interpretation on whether a no-follow-on strategy actually improves returns.

Hypothesis 2: Small Investments Have Better Returns than Large Ones

Another common VC mantra is that you should "put a lot of money to work" in each investment.  To me, this strategy seems more like a way to reduce transaction costs than improve outcomes, which is fine, but the distinction is important.  Smaller investments probably occur earlier so they should be higher risk and thus higher return.  Also, if everyone is trying to get into the larger deals,  smaller investments may be less competitive and thus offer greater returns.I chose $300K as the dividing line between small and large investments, primarily because that was our  original forecast of average investment for RSCM (BTW, we have revised this estimate downward based on recent trends in startup costs and valuations).  The Excel file with 399 records of "small" investments is here.  The IRR was 39% and the payout multiple was 4.0x.  Again, a huge edge over the entire sample!  Interestingly, less of an edge in IRR but more of an edge in multiple than the no-follow-on test.  But smaller investments may take longer to pay out if they are also earlier.  IRR really penalizes hold time.Interesting side note.  When I backtested the RSCM strategy, I keyed on investment "stage" as the indicator of risky early investments.  Seeing as how this was the stated definition of "stage", I thought I was safe.  Unfortunately, it turned out that almost 60% of the records had no entry for "stage".  Also, many of the records that did have entries were  strange.  A set of 2002 "seed" investments in one software company for over $2.5M?  A 2003 "late growth" investment in a software company of only $50K?  My guess is that the definition wasn't clear enough to investors filling out the survey.  But I had committed to my hypothesis already and went ahead with the backtest as specified.  Oh well, live and learn.

Hypothesis 3: Post-Crash Returns Are No Different than Pre-Crash Returns

As you probably remember, there was a bit of a bubble in technology startups that popped at the beginning of 2001. You might think this bubble would make angel investments from 2001 on worse.  However, my guess was that returns wouldn't break that cleanly.  Sure, many 1998 and some 1999 investments might have done very well.  But other 1999 and most 2000 investments probably got caught in the crash.  Conversely, if you invested in 2001 and 2002 when everybody else was hunkered down, you could have picked up some real bargains.The Excel file with 168 records of investments from 2001 and later is here.  23% IRR and 1.7x payout multiple.  Ouch!  Was I finally wrong?  Maybe.  Maybe not.  The first problem is that there are only 168 records.  The sample may be too small.  But I think the real issue is that  the dataset "cut off" many of the successful post-bubble investments because it ends in 2007.To test this explanation, I examined the original AIPP data file.  I filtered it to include only investment records that had an investment date and where time didn't run backwards.  That file is here.  It contains 304 records of investments before 2001 and 344 records of investments in 2001 or later.  My sample of exited investments contains 284 records from before 2001 and 168 records from 2001 or later.  So 93% of the earlier investments have corresponding exit records and 49% of the later ones do.  Note that the AIPP data includes bankruptcies as exits.So I think we have an explanation.  About half of the later investments hadn't run their course yet.  Because successes take longer than failures, this sample over-represents failures.  I wish I had thought of that before I ran the test!  But it would be disingenuous not to publish the results now.

Conclusion

So I think we've answered some interesting questions about angel investing.  More important, the process demonstrates why we need to collect much more data in this area.  According to the Center for Venture Research, there are about 50K angel investments per year in the US.  The AIPP data set has under 500 exited investments covering a decades long span.  We could do much more hypothesis testing, with several iterations of refinements, if we had a larger sample.

analyzing subsets of the AIPP dataneed

In my work for RSCM, one of the key questions is, "What is the return of angel investing?"  There's some general survey data and a couple of angel groups publish their returns, but the only fine-grained public dataset I've seen comes from Rob Wiltbank of Willamette University and the Kauffman Foundation's Angel Investor Performance Project (AIPP).

In this paper, Wiltbank and Boeker calculate the internal rate of return (IRR) of AIPP investments as 27%, using the average payoff of 2.6x and the average hold time of 3.5 years.  Now, the arithmetic is clearly wrong: 1.27^3.5 = 2.3.  The correctly calculated IRR using this methodology is 31%. DeGenarro et al report (page 10) that this discrepancy is due to the fact that Wiltbank and Boeker did not weight investments appropriately.

In any case, the entire methodology of using average payoffs and hold times is somewhat iffy.  When I read the paper, I immediately had flashbacks to my first engineering-economics class at Stanford.  There was a mind-numbing problem set that beat into our skulls the fact that IRR calculations are extremely sensitive to the timing of cash outflows and inflows.  I eventually got a Master's degree in that department, so loyally adopted IRR sensitivity as a pet peeve.

To calculate the IRR for the AIPP dataset, what we really want is to account for the year of every outflow and inflow.  The first step is to get a clean dataset.  I started by downloading the public AIPP data.  I then followed a three step cleansing process:

1. Select only those records that correspond to an exited investment.
2. Delete all records that do not have both dates and amounts for the investment and the exit.
3. Delete all records where time runs backwards (e.g., payout before investment).

The result was 452 records.  A good-sized sample.  The next step was to normalize all investments so they started in the year 2000.  While not strictly necessary, it greatly simplified the mechanics of collating outflows and inflows by year.  Finally, I had to interpolate dates in two types of cases:

1. While the dataset includes the years of the first and second follow on investment, it does not include the year for the "followxinvest". For the affected 12 records, I interpolated by calculating the halfway point between the previous investment and the exit, rounding down.  Note that this is a conservative assumption.  Rounding down pushes the outflow associated with the investment earlier, which lowers the IRR.
2. For 78 records, there are "midcash" entries where investors received some payout before the final exit.  Unfortunately, there is no year associated with this payout.  A conservative assumption pushes inflows later, so I assumed that the intermediate payout occurred either 0, 1, or 2 years before the final exit.  I calculated the midpoint between the last investment and the final exit and rounded down.  If it was more than 2 years before the final exit, I used 2 years.

With these steps completed, I simply added up outflows and inflows for every year and used the Excel IRR calculation.

The result was an IRR of 30% and a payoff multiple of 2.4x with an average hold time of 3.6 years.

Please note that this multiple is slightly lower than the 2.6x and the hold time is slightly higher than the 3.5 years Wiltbank and Boeker calculated for the entire dataset.  Thus, my results do not depend on accidentally cherry-picking high-returning, quick-payout investments.  If you want to double-check my work, you can download the Excel file here.

All in all, a satisfying result.  Not too different from what's other people have published, but I feel much more confident in the number.  For anyone analyzing subsets of the AIPP data, I've found that my Excel file makes it pretty easy to calculate those returns.  Just zero out all records you don't care about by selecting the row and hitting the "Delete" key.  The return results will update correctly.  But don't do a "Delete Row".  Then a bunch of the cell references will be broken.  

[Update 1/27/11: I've done a follow up post on using this method to test various hypotheses.]

Jeff Miller has done a couple of nice posts on "A Simulation of Angel Investing" here and here. I think it's terrific that Jeff actually asked the question and tried to answer it with simulation. However, his answer of 20 is way too low because of two key oversimplifications. Using a more sophisticated methodology, I'll show that a better answer is 100 to 150.You may recall that Saving the World with Startups explained the "why" of RSCM. Our goal is to increase the number of technology startups. In some sense, this post describes the "how". Well, at least part of it. One of the biggest barriers to getting a company off the ground is finding working capital. Ergo, we need to figure out how to facilitate investments in startups. More precisely, we need to promote seed-stage investments because those are what help founders initially launch their companies.The ideal solution would be an investment vehicle that can turn huge chunks of money into digestible seed-stage bites with a return that induces plenty of investors to participate. But here are some slightly scary statistics. 50% of all seed-stage startups fail and returns come disproportionately from the top 10%. As all you poker players in the audience will note, you're making big bets with high variance. The natural question is, "How many bets should you place?"To answer this question, I've built several generations of seed-stage investing simulations for RSCM. My models are rather complicated because we wanted to evaluate a bunch of secondary questions such as whether it's better to do follow on investments, what happens if the balance between seed and Series A valuations changes, and what happens in cases where a startup does poorly initially but then takes off.  Therefore, I actually had to model the startup lifecycle round by round and the mechanics became very complex. (If you're not a quant, you can stop reading now.  Things are going to get real geeky real fast).However, a simplified single-round version of my model will illustrate the missing pieces of Jeff's model. The first is what diversification means. He focuses on the risk of total loss and the chances of not getting at least one "hit". In my opinion, the question you really want to ask is what the probability is that you'll under-perform the market by more than a given amount. For example, what's the probability that you'll under-perform by more than 25%?  The logic here is that you invest in an asset class because of the overall return of that asset class, so you want to know the chances that you'll realize returns in that ballpark.The second key oversimplification is that Jeff uses a discrete probability distribution of returns. If you've read Taleb's The Black Swan, you know this is a mistake because at least some seed-stage outcomes probably follow a Pareto distribution. The key characteristic of this distribution is that regions of extreme outcomes are self similar.  So not only do the top 10% of companies represent a disproportionate share of the returns, the top 10% of the top 10% represent a disproportionate share of those returns.  And so on.  And so on.  20 investments may be enough to get you a fair share of the top 10%, but not enough to get you a fair share of the top 1%.So here's my simplified model, which roughly follows Jeff's qualitative taxonomy:

• 50% failures: the company utterly fails. The investor gets 0 money returned.
• 20% break even: the company achieves some limited success and the money returned follows a lognormal distribution with a minimum of 0, a, mean of 1, and a standard deviation of 1.  So an average outcome is 1.0x and 1 standard deviation above is 2.0x.
• 20% decent: the company achieves substantial success and the money returned follows a lognormal distribution with a minimum of 2, a mean of 4, and a standard deviation of 4. So the minimum outcome is 2x, the mean outcome is 4x, and 1 standard deviation above is 8x.
• 10% homeruns: the company achieves massive success and the money returned follows a Pareto distribution with a location of 10 and an index of 1.5. So the minimum outcome is 10x and the mean outcome is 30x.

Now, we can compute the expected value of an investment as .50*0 + .2*1 + .2*4 + .1 *30 = 4.0.  The data I've seen puts the average hold time for successful angel investments at 6 years, so this would imply an IRR of about 26%. This is in line with the available research on angel returns (RSCM has a summary of this research here).I ran a simulation with these parameters using Oracle's Crystal Ball, producing an overall return distribution for a run of 100K trials. Here's the excess distribution plot (the probability that the money returned will exceed a given multiple), truncated at 50x for some semblance of readability:

The return across the entire simulation was 4.05x (very close to the analytically expected return of 4.0x).  The maximum return was 8,361x (think Andy Bechtolsheim's $100K investment in Google which was eventually worth about $1B).  The top 10% accounted for 77% of the total return.  The top 1% accounted for 35%.  The top .1% accounted for 17%. We can already see that a portfolio of 20 will be insufficient.The source file is AngelSimulation. Most of you probably don't have Crystal Ball so this will look like a pretty useless Excel file to you.  However, I set up the run to output just the AngelSimulationData in an Excel file.  Anyone can analyze this with standard charting tools or import the data for use by his own code.I've also got another AngelSimulationPortfolios with a macro that generates 10K random portfolios of a given size from the trial data.  I've run it for portfolio sizes from 10 to 200 in intervals of 10. After sorting the portfolio returns at the specified size, the macro calculates the probability of hitting 75% of the market return by seeing what percentage of the portfolio returns are greater than 3.0. Here's a chart of those probabilities:

[Edited 5/14 in response to suggestion from AN]. As you can see, 20 investments isn't nearly enough if you're a fund investing other people's money.  Worse than a coin flip that you'll hit 75% of the market return. In fact, in my simulated portfolio data, there's about a 7% chance that you'll lose money with a portfolio of 20 investments.  Personally, I'd say you want a fund to be in the 100 to 150 investment range.  But it's different for individual investors putting in their own money.  I'd say you want to hit at least a 50% chance of realizing 75% of the market return, which would be 30 investments.  Now, if you think you think you have some forecasting skill and less than 50% of your seed investments will fail and/or more than 10% will be homeruns, 20 may be plenty.Of course, if you accept the thesis that 100-150 is the right range for a fully diversified fund-like portfolio, you may now be asking yourself how making that many seed-stage investments  is logistically possible. The challenge is actually worse than that.  Due to vintage risk, you probably want to make 100-150 investments per year or at least every few years. But that's a story for another day...

[EDITED 05/08/2009: see here] The majority of people I've talked to like the idea of revolutionizing angel funding. Among the skeptical minority, there are several common objections. Perhaps the weakest is that individual angels can pick winners at the seed stage.

Now, those who make this objection usually don't state it that bluntly. They might say that investors need technical expertise to evaluate the feasibility of a technology, or industry expertise to evaluate the likelihood of demand materializing, or business expertise to evaluate the evaluate the plausibility of the revenue model. But whatever the detailed form of the assertion, it is predicated upon angels possessing specialized knowledge that allows them to reliably predict the future success of seed-stage companies in which they invest.

It should be no surprise to readers that I find this assertion hard to defend. Given the difficulty in principle of predicting the future state of a complex system given its initial state, one should produce very strong evidence to make such a claim and I haven't seen any from proponents of angels' abilities. Moreover, the general evidence of human's ability to predict these sorts of outcomes makes it unlikely for a person to have a significant degree of forecasting skill in this area.

First, there are simply too many random variables. Remember, startups at this stage typically don't have a finished product, significant customers, or even a well-defined market. It's not a stable institution by any means. Unless a lot of things go right, it will fall apart. Consider just a few of the major hurdles a seed-stage startup must clear to succeed.

1. The team has to be able to work together effectively under difficult conditions for a long period of time. No insurmountable personality conflicts. No major divergences in vision. No adverse life events.
2. The fundamental idea has to work in the future technology ecology. No insurmountable technical barriers. No other startups with obviously superior approaches. No shifts in the landscape that undermine the infrastructure upon which it relies.
3. The first wave of employees must execute the initial plan. They must have the technical skills to follow developments in the technical ecology. They must avoid destructive interpersonal conflicts. They must have the right contacts to reach potential early adopters.
4. Demand must materialize. Early adopters in the near term must be willing to take a risk on an unproven solution. Broader customers in the mid-term must get enough benefit to overcome their tendency towards inaction. A repeatable sales model must emerge.
5. Expansion must occur. The company must close future rounds of funding. The professional executive team must work together effectively. Operations must scale up reasonably smoothly.

As you can see, I listed three example of minor hurdles associated with each major hurdle. This fan out would expand to 5-10 if I made a serious attempt at exhaustive lists. Then there are at least a dozen or so events associated with each minor hurdle, e.g., identifying and closing an individual hire. Moreover, most micro events occur repeatedly. Compound all the instances together and you have an unstable system bombarded by thousands of random events.

Enter Nassim Taleb.  In Chapter 11 of The Black Swan, he summarizes a famous calculation by mathematician Michael Berry: to predict the 56th impact among a set of billiard balls on a pool table, you need to take into account the the position of every single elementary particle in the universe.  Now, the people in a startup have substantially more degrees of freedom than billiard balls on a pool table and, as my list above illustrates, they participate in vastly more than 56 interactions over the early life of a startup. I think it's clear that there is too much uncertainty to make reliable predictions based on knowledge of a seed-stage startup's current state.

"Wait!" you may be thinking, "Perhaps there are some higher level statistical patterns that angels can detect through experience." True. Of course, I've poured over the academic literature and haven't found any predictive models, let alone seen a real live angel use  one to evaluate a seed stage startup. "Not so fast! " you say, "What if they are intuitively identifying the underlying patterns?" I suppose it's possible.  But most angels don't make enough investments to get a representative sample (1 per year on average).  Moreover, none of them that I know systematically track the startups they don't invest in to see if their decision making is biased towards false negatives. Even if there were a few angels who cleared the hundred mark and made a reasonable effort to keep track of successful companies they passed on, I'd still be leery.

You see, there's actually been a lot of research on just how bad human brains are at identifying and applying statistical patterns. Hastie and Dawes summarize the state of knowledge quite well in Sections 3.2-3.6 of Rational Choice in an Uncertain World. In over a hundred comparisons of human judgment to simple statistical models, humans have never won.  Moreover, Dawes went one better. He actually generated random linear models that beat humans in all the subject areas he tried. No statistical mojo to determine optimal weights. Just fed in a priori reasonable predictor variables and a random guess at what their weights should be.

Without some sort of hard data amenable to objective analysis, subjective human judgment just isn't very good. And at the seed stage, there is no hard data. The evidence seems clear. You are better off making a simple list of pluses and minuses than relying on a "gut feel".

The final line of defense I commonly encounter from people who think personal evaluations are important in making seed investments goes something like, "Angels don't predict the success of the company, they evaluate the quality of the people. Good people will respond to uncertainty better and that's why the personal touch yields better results." Sorry, but again, the evidence is against it.

This statement is equivalent to saying that angels can tell how good a person will be at the job of being an entrepreneur. As it turns out, there is a mountain of evidence that unstructured interviews have little value in predicting job performance. See for example, "The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings" [EDITED 10/17/2011: New link to paper because old one was stale]. Once you have enough data to determine how smart someone is, performance on an unstructured interview explains very little additional variance in job performance. I would argue this finding is especially true for entrepreneurs where the job tasks aren't clearly defined. Moreover, given that there are so many other random factors involved in startup success than how good a job the founders do, I think it's hard to justify making interviews the limiting factor in how many investments you can make.

Why then are some people so insistent that personal evaluation is important?  Could we be missing something? Always a possibility, but I think the explanation here is simply the illusion of control fallacy. People think they can control random events like coin flips and dice rolls. Lest you think this is merely a laboratory curiosity, check out the abstract from this Fenton-O'Creev, et al study of financial traders. The higher their illusion of control scores, the lower their returns.

I'm always open to new evidence that angels have forecasting skill. But given the overwhelming general evidence against the possibility, it better be specific and conclusive.