In an earlier post I wrote about risk-driven development. The idea that I proposed was to address project risks starting with the biggest risk first (to “fail early” if you have to fail). In this post I will try to elaborate on that rather vague statement and prove that a strategy along these lines indeed maximizes the value of the project.

The traditional way to evaluate projects is by using discounted cash flow valuation (DCF). The model assumes that you make a big one off investment and then get a positive cash flow from that investment. This model works fine if we are investing in say a new paper machine, provided that we can foresee the future cash flow attributable to the new machine. The risk of the investment is weighted in by discounting the future cash flow with a discount factor that is a function of the risk level.

In new product development projects that use the Stage-Gate model we get to make incremental decisions and thus do the investment piecewise while learning along the way. (It’s not as easy to see how one can learn from investing in say a 10:th of a paper machine.) If the market changes or we realize that we can’t overcome a technical challenge then we can abort the whole project minimizing our losses before we’ve spent all our money. This possibility to make incremental decisions increases the expected value of the project by decreasing the expected cost. This is the principle behind real options valuation.

Another way to see Stage-Gate is as a series of consecutive go-no go experiments. Each successful experiment takes us one step closer to the full product. If the experiment fails then we abort the project. All experiments must succeed in order for the whole project to succeed.

Let’s look more closely at the stages in a Stage-Gate (project) process: We do some work in each stage and based on the results we either abort the project with probability

$1–{p}_{1}$

or continue the project with probability

${p}_{1}$

We abort the project if a fatal (for the project) risk has been realized during the stage. The probability to abort is therefore here equal to the probability of a fatal risk being realized.

The question discussed in many of the posts on this blog is: in what order should we do the experiments in order to maximize the value of the project? For this we need to introduce the concept of a decision tree and some associated entities.

 A number of consecutive experiments represented as a decision tree.

In the decision tree we have a number of “events” depicted as circles. These represent our experiments. Each experiment has a cost of

${c}_{i}$

,

a probability of succeeding of
${}_{}$
p
i

, and a probability of failing of
$1$

p
i

. The cost of failing an experiment is
${}_{}$
C
i

,
${}_{}$
C
i

and
${}_{}$
c
i

>
0

. Also

${}_{}$
C
i

=

n
=
1

i

cn

which means that the cost of failing the project at the n:th experiment (by failing the n:th experiment) is the accumulated cost of all experiments up to and including the n:th.

We can “unfold” the value of the project step by step. Let’s look at the value
${}_{}$
V
1

of the project before the first experiment. It is simply the probability weighted average of the value of the two branches.

${}_{}$
V
1

=
(
1

p
1

)
(
&InvisibleTimes;

C

1

)
+

p
1

&InvisibleTimes;
(

C

1

+

V
2

)
=

C

1

+

p
1

V
2

If the first experiment fails we will have a negative value
$V$
=

C

1

=

c

1

of the first experiment. Otherwise we get whatever comes down the other branch which is
${}_{}$
V
2

subtracted with the cost
${}_{}$
C
1

.

${}_{}$
V
2

,
${}_{}$
V
3

, and
${}_{}$
V
4

can be written in the same format.

${}_{}$
V
2

=

C

2

+

p
2

V
3

${}_{}$
V
3

=

C

3

+

p
3

V
4

${}_{}$
V
4

=

C

4

+

p
4

I
${}_{}$
V
4

is where it gets a little more interesting as it is here we actually have an opportunity to get some income
$I$

.

Untangling the recursion we get

$V$
=

V
1

=

C

1

p
1

C
2

p
1

p
2

C
3

p
1

p
2

p
3

C
4

+

p
1

p
2

p
3

p
4

I

The income
$I$

is multiplied by all probabilities so for the income the order of the experiments doesn’t matter. Maximizing the value with respect to the order of the experiments is therefore equivalent to minimizing the cost (remember that all costs in the expressions here have positive values). So we need to minimize

$C$
=

C
1

+

p
1

C
2

+

p
1

p
2

C
3

+

p
1

p
2

p
3

C
4

with respect to the order of the experiments with costs
${c}_{j}$

, and associated probabilities for success
${p}_{j}$

. It is also from the above easy to guess what the expression for the cost is with an arbitrary number of experiments. I choose intuition before induction for now though and will not try to prove it.

What we want is a rule or a set of rules for sorting the experiments so as to minimize the expected cost. Let’s first assume that the order of the experiments
${}_{}$
E
i

as shown in the figure above minimizes the total cost
$C$

. Any permutation of the experiments would therefore increase the cost. From this we can deduce how the
${c}_{i}$

and
${p}_{i}$

must relate to each other.

Now trade places between the first and the second experiment. This should (per definition) give a higher expected cost. Expanding all
${C}_{i}$

into their constituencies and setting up the inequality we get

${}_{}$
c
1

+

p
1

(

c
1

+

c
2

)
+

p
1

p
2

(

c
1

+

c
2

+

c
3

)
+

p
1

p
2

p
3

(

c
1

+

c
2

+

c
3

+

c
4

)
<

c
2

+

p
2

(

c
2

+

c
1

)
+

p
2

p
1

(

c
2

+

c
1

+

c
3

)
+

p
2

p
1

p
3

(

c
2

+

c
1

+

c
3

+

c
4

)

After some juggling around we finally get

${}_{}$
c
1

+

p
1

(

c
1

+

c
2

)
<

c
2

+

p
2

(

c
1

+

c
2

)

Switching any two adjacent experiments give similar (but not entirely the same) inequalities

${}_{}$
c
2

+

p
2

(

c
1

+

c
2

+

c
3

)
<

c
3

+

p
3

(

c
1

+

c
2

+

c
3

)

and

${}_{}$
c
3

+

p
3

(

c
1

+

c
2

+

c
3

+

c
4

)
<
c
4

+

p
4

(

c
1

+

c
2

+

c
3

+

c
3

)

As long as all inequalities above are true, we will increase the cost by reversing the order of two adjacent experiments. I have not managed to prove that the pair-wise inequalities are a sufficient condition for a global minimum. Switching the first and the third experiment would for instance give the inequality

${}_{}$
c
1

+

p
1

&InvisibleTimes;
(

c
1

+

c
2

)
+

p
1

&InvisibleTimes;

p
2

&InvisibleTimes;
(

c
1

+

c
2

+

c
3

)
<

c
3

+

p
3

&InvisibleTimes;
(

c
2

+

c
3

)
+

p
2

&InvisibleTimes;

p
3

&InvisibleTimes;
(

c
1

+

c
2

+

c
3

)

which doesn’t necessarily follow from the pair-wise inequalities above it. Remains also to do the math for an arbitrary number of experiments but that seems like the easier of the two remaining issues.

The expressions in the inequalities are easy enough to put in a spreadsheet to get simple tool for ordering a number of experiments though. I did just that and the spreadsheet simulation show that the conditions above are a predictor for a global minimum with the admittedly small number of experiments I have carried out. I therefore still dare to postulate that we wish to have a small
${}_{}$
c
i

in some way combined with a small
${}_{}$
p
i

in early experiments. Remember that
${}_{}$
p
i

is the probability of succeeding with the experiment. A small probability of success means a large probability of failure means that we should do the uncertain and cheap experiments to start with.

The spreadsheet simulation I did for instance gives that if we have a series of four experiments with costs 20, 30, 40, and 20 with the corresponding probabilities for success of 0.4, 0.6, 0.8, and 0.9, then we should order the experiments in the order 1, 2, 4, 3 whereby we get an expected cost of 80.56. The sum of the costs of all experiments is 110 so by doing the experiments one at the time and aborting if failing we can bring down our expected cost by 27%. With many other random ways to order the experiments we will only decrease or expected cost by a few percent.

In conclusion: the riskier the project, the more we will gain (a) by using some kind of Stage-Gate model with a decision to continue or to abort after each experiment (or group of experiments) and (b) by ordering the experiments with those that give most uncertainty reduction for the money in the beginning.

When I started this post I was hoping that either the proof would be pretty easy (there is after all no esoteric mathematics involved) or that it would fall into a class of well-known problems such as a shortest path or a traveling salesman that already have solutions. But so far, no luck. I will keep on looking and if you, dear reader, have some ideas, please let me know. Until then, I’m going to trust my hunch and my incomplete proof.