Robin Karol, CEO of the PDMA once told me “By far the biggest mistake in decision making is treating an evaluation as an analysis rather than a conversation.” This view is a bit astonishing, as many analysts spend a lot of time learning about the mathematical modeling in evaluation. It is a trap I have fallen into.
My conversation with Robin brought a flashback to early in my career. I saw myself in a boardroom finishing the presentation of what I thought was a brilliant analysis to the executives. The bottom line: closing the power plant early would save the company tons of money. There were a few questions, easily answered from my analysis. Then one executive piped up, “I don’t know, my gut says otherwise.”
Shocked, I marshaled my wits and prepared to use the analysis to convince the skeptic. But I did not have the chance. The president politely dismissed my colleagues and I, “Thank you for your hard work. We will take it under advisement.”
As we left the boardroom, my colleague offered me these consoling words, “Don’t worry David, you did a good job. Lots of decisions get made on gut.”
These were empty words, consolation for wasted analysis. I had made Robin’s number one mistake.
This mistake shows up in other ways as well. People treat evaluations as boxes to be checked on some process form so they can get on with the real work. People use evaluations to persuade investors by making outrageous assumptions and saying “the analysis says go!” with a straight face. People use evaluations to make points in arguments, or to negotiate a budget. In all of these cases, the evaluation is merely an analysis—a tool or a deliverable in a mathematical format.
So what?
From an analyst’s point of view, this is frustrating because it invalidates lots of hard work. From a business point of view, this is inefficient because the hard work is essentially wasted effort for all the participants. But fundamentally, the damage is more than frustration or waste: it represents a failure to learn. In complex or unfamiliar situations, it is impossible to make good judgments uniformed by good evaluations. This failure to learn leads to poor decisions and poor performance in many organizations.
In this case, according to my analysis, failure to close the power plant could lead to catastrophic loss for the company. Their major city the utility served was discussing dropping the company because costs were too high. Yet it seemed that the executives were not getting the message.
While these dysfunctions may seem like part of what it means to be in an organization, a root cause analysis shows it does not have to be this way.
To treat an evaluation as a conversation, you need to start with questions people actually have and craft a model to answer them, in a spirit of inquiry. You need to provide variables that reflect people’s concerns and allow them to express themselves. You need to use analytical results and models as a forum for learning about what matters and the best course of action.
In ongoing conversations, Robin and I shaped a list of the biggest mistakes in evaluation. They all flow this basic insight that you must treat an evaluation as a conversation, not an analysis. Here they are in brief:
1. PEOPLE – treating a project evaluation as an analysis rather than as a conversation. Not having the right people (or process) to have the conversation. Failure to connect the analysis to the decision making process.
2. TOOLS – Use of convenient tools that lack the power to provide real insight or drive a lasting business decision.
3. VALUE – Failing to bring an evaluation to the bottom line prevents real discussions of business issues and tradeoffs. Narrow view of value as NPV. Undocumented assumptions lead people to believe that a particular value metric is more comprehensive than warranted.
4. UNCERTAINTY – Critical issues covered up using assumptions rather than explicitly quantified as uncertainty. Failure to evaluate, vet and learn about uncertainty. No understanding of the business’ ability to manage risk and uncertainty.
Fortunately for my earlier self, the power plant decision was not made in that boardroom on that day. Somehow we were able to correct for the mistakes that Robin and I later identified.
We followed up with the skeptical executive and tried to understand his gut. After some inquiry, we realized that he had an implicit assumption: that decommissioning costs would be lower in the future. Other executives where concerned these costs would be high. They were all making assumptions, and failing to see this issue as an uncertainty.
We repurposed the analysis to show that both were in some sense right in their gut: if decommissioning costs were low, then it really was better to keep the plant open; if they were high it was better to close it now. What was at issue was the probability of high or low decommissioning costs. Both sides confessed to having little expertise in this area. By connecting the uncertainty to value, we were able to bring both sides together.
Of all the concerns, politics, debates and confusion surrounding the plant decision, the best course of action boiled down to one key thing: decommissioning costs. This alone was a huge advance in the conversation because the executive team stopped debating issues that did not matter. The tools we were using had the power to focus the discussion productively.
A quick study was commissioned and the evidence for rising decommission costs was strong. The skeptical executive himself assigned a small chance that costs would remain low, and was persuaded to close the plant. He, and the whole executive team, had learned and now had a more informed “gut”. The conversation had been guided effectively by the evaluation.
A couple of months later, they sent me a newspaper clipping. It featured a picture of the skeptical executive in front of the power plant. He was announcing its closure. I should really show it to Robin one day.
When the new R&D Chief arrived, he saw his duty to make the hard calls.
Recent Comments