Evaluating and Scoring Companies: Analyst vs. Crowd vs. Programmatic

External company evaluations and scores can help buyers and dealmakers narrow the field when assessing vendors or deal prospects, especially as the pool of available opportunities only grows.

I see 3 approaches being used, and will offer my thoughts on each method below:

  1. Analyst evaluation
  2. Crowd evaluation
  3. Programmatic evaluation

1) Analyst evaluations are the bailiwick of research shops whose analysts look at a sector and use a specific evaluation rubric to classify companies. For example, Gartner uses the magic quadrant methodology, where analysts evaluate companies based on completeness of vision on the X-axis and the ability to execute on the Y-axis. They then group companies into leaders, challengers, niche players, or visionaries (see figure below).

  • Pros: Analysts conduct primary research, talk to companies, and dig through operational details to synthesize a large amount of non-public data to form opinions.
  • Cons: Analyst evaluations can be subjective and biased based on the relationship a vendor has with the analyst and can also be very limited as analysts do not have the bandwidth to cover all the up and coming entrants in any given sector. 

Screen Shot 2014-08-22 at 1.12.23 PM

2) Crowd evaluations use the power of the open web to enable the wisdom of the crowds to score individual companies. For example, ProductHunt uses a simple Reddit like model that allows their community to list and up-vote companies. Companies receiving the lion’s share of up-votes in any given sector are the presumed leaders in a particular sector.

  • Pros: Crowd evaluations democratize scoring and can signal which companies have a high likelihood of pleasing customers. 
  • Cons: Crowd evaluations are broad in nature and don’t determine the fit of a company against specific customer needs, nor do they evaluate a company based on their operations and strategy.

3) Programmatic evaluations use algorithms that take in available data from the web to determine an overall score for a particular company. For example, Mattermark uses week-to-week growth data on a company’s web traffic, mobile downloads, inbound links, Twitter followers, Facebook page likes, and LinkedIn followers to arrive at an overall score for the company.

  • Pros:  Programmatic approaches that leverage available public data can be a good signal for companies that are showing growth and can help buyers and investors find up and coming opportunities.
  • Cons: The flip side is that programmatic approaches can favor socially trending companies versus companies that have a stable offering and an existing base of customers. Furthermore, they do not evaluate companies based on any specific customer needs, nor do they take into account a company’s overalls strategy and ability to execute.

Ultimately, each of these approaches provides part of the solution, and at Venture Scanner we believe the final answer lies at the intersection of all these approaches. Our approach of “analyst coverage as a service” combines a sector analyst’s primary research with aggregated and synthesized data from the web to provide our customers with the necessary insights to make a decision

Are there other approaches or pros/cons that I’m missing?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s