While I am obviously disappointed with the outcome, the BCS pairings were mostly free from controversy this year. The co
VIEWPOINT BCS Rankings and New Product Development Scoring Systems A computer does not adequately account for heart and character BY BILL POSTON
So my Longhorns lost the Bowl Championship Series (BCS) National Championship game to a team named after an algae bloom. Not that I am bitter, but I believe that Sandra Bullock deserves to win Best Actress for pretending to be all hot and bothered when meeting Nick Saban in The Blind Side. I don’t like that guy and neither does she. While I am obviously disappointed with the outcome, the BCS pairings were mostly free from controversy this year. The convoluted system that has been continuously tweaked since its inception in 1998 seemed to work. The complex algorithm that underlies the ranking methodology uses a series of Borda counts and combines the results of a sports writers poll, a coaches poll, and an average of six different computer scoring systems to arrive at its overall rankings (Borda was an eighteenth century French mathematician who didn’t know anything about college football). Football people don’t really trust computers – or the French – so the human polls are now weighted more heavily. But the only reason we have computer scoring systems is because we were, once upon a time, not happy with the consistency of the results of the human polls. We have the LSU version of Nick Saban to thank for that. The benefit of the computer ranking systems is that they use objective criteria to evaluate the performance of teams and that they are not subject to “reputation bias” or any other bad ol’ form of subjective selection by mere human beings. I think that having six different computer models account for a third of the input is about right for college football and they appear to be doing a decent job of predicting success. No one really thought that Cincinnati was going to beat Florida. Did they? Predicting success is also the objective of scoring systems used by companies in the new product development process. I have worked with dozens of clients that have extremely sophisticated systems for scoring and ranking new product concepts. These systems identify characteristics of new product projects that are historically correlated with winners and use these measures as predictors of success.
The problem occurs when these algorithmdriven systems become the de facto decision making tool. Even the BCS gives the computer only onethird of the vote. If Excel can tell us which new product concepts we should invest in then why do we need Vice Presidents?
VIEWPOINT I like these systems and believe that they are, if properly constructed, a valuable input into the project selection and portfolio prioritization process. The problem occurs when these algorithm-driven systems become the de facto decision making tool. Even the BCS gives the computer only one-third of the vote. If Excel can tell us which new product concepts we should invest in then why do we need Vice Presidents? When we are talking about innovation we need to first consider that we are dealing with tremendous amounts of uncertainty. No matter how hard we try to quantify variables and assign probabilities to projections, we are still dealing with forecasts and estimates. Just because we put these numbers into a fancy algorithm doesn’t make them true. We can get better over time by tweaking the algorithm based on demonstrated in-market success, but we will always have to deal with – and embrace – uncertainty. That is where management judgment comes into play. There is no good way to automate project selection and prioritization. Scoring systems should be an input into decision makers that aggressively question the assumptions behind the model and parse the variables that go into it. In my opinion, an informed executive with a good gut beats a BASS (big-assed spreadsheet) every day. Give the computer a voice but don’t let it set your priorities. The best models out there are not perfect predictors of success and cannot replace a seasoned executive’s knowledge of, and feel for, the marketplace. So let’s make sure that our scoring models are used properly and improved over time. The BCS ranking system will continue to evolve and I’ll eventually get over the Longhorns’ loss. The computer does not adequately account for heart and character. If it did, Texas would be the clear #1. Hook ’em Horns!
KALYPSO CONTACT
Bill Poston, Managing Partner
[email protected] www.kalypso.com
Give the computer a voice but don’t let it set your priorities. The best models out there are not perfect predictors of success and cannot replace a seasoned executive’s knowledge of, and feel for, the marketplace.
