By Egil H. Lehmann and Hogne Sandvik.
Department of Public Health and Primary Health Care, Division for General Practice, University of Bergen, Norway.
The purpose of the present article is to show how computer simulation can enable clinicians with only a limited background of statistics to design experiments of the so-called sequential type.
In ordinary, 'classical' experimentation, observations are examined when the experiment itself is finished. In sequential experiments, the observations are examined in a systematic manner as the experiment proceeds, in order to terminate the experiment at the earliest possible time.
Sequential experimentation is attractive from many points of view. In medicine, ethical and economical arguments weight heavily in favour. Also, it is a fact of life that many experiments said to be classical are in fact of the sequential type, because researchers allow themselves informal analyses of the results before the experiment is finished. Would it not be nicer to include these 'looks' into the formal design?
Beginning in 1960, a large literature has grown around sequential experimentation in medicine (1,2,5). However, the majority of medical experiments are still of the classical kind. Evidently bemused by this, pioneers have listed various disadvantages that may have kept researchers away from sequential plans (3). Sequential plans lead to more imprecise estimates, they may be difficult to administrate, they may not be sufficiently known, they may have gotten a reputation of being theoretically difficult.
We suspect the latter. Medical studies, aimed at creating good clinicians, even today are almost devoid of statistical theory. The minimum of statistics necessary to do research is learnt in post-graduate courses, which may make the candidates able to design classical experiments with a specified power. Proceeding a step further, to designing sequential experiments, would require months of further study, time which, for an aspiring clinician, simply is not there.
DIAGRAM PLANS AND R.S.T. (GROUP SEQUENTIAL) PLANS
The first sequential plans were presented in the form of diagrams, where observations could be plotted as they came, and where the crossing of a borderline lead to termination of the experiment. Later, Armitage introduced the R.S.T (Repeated Statistical Tests) plans, replacing the borderlines with repeated Student's 't' tests (1,4,5).
It seemed natural to perform the 'interim' tests at regular intervals, by each time adding an equally large group of new observations to the observations already collected. Hence, the term 'group sequential plans'.
The Level of significance nearly always preferred in classical experiments, 0.05, should also be the preferred total level of a group sequential plan. To achieve this, each single interim test, as well as the eventuel final one, must be performed on a stricter Level of significance. These levels have been tabulated by Armitage and Klim McPherson (Table 1).
One would also need to calculate the sample sizes necessary to obtain a certain power. In spite of the fact that the size of a sequential plan is unknown beforehand, one may specify a certain maximal size at which the experiment shall be terminated regardless of the observations. The calculation of this maximal sample size is not elementary, but it may be easily found by stochastic computer simulation. Tables based on the normal distribution are presented by Pocock (5).
STOCHASTIC SIMULATION OR 'MONTE CARLO' TECHNIQUE
In the beginning, statistical teory was established to calculate the results of chance games. First were the games, then came the theory to 'explain' them. The games and the theory were regarded as proof of each other. Today, when statistical theory has become an established part of mathematics, games are used in cases where the mathematical solution is difficult to achieve. Hence the expression 'Monte Carlo technique'.
Today, these games are played on computers. The computer is made to produce numbers in a semirandom fashion. By adjusting and sifting these numbers, they can be made to imitate observations.
To simulate sequential experiments, a computer program is needed that performs three tasks: Producing data resembling the observations from specified experiments, performing statistical tests at the right places, and keeping books of the results so that the proportion of tests deemed 'significant' can be calculated.
The proportion of tests deemed 'significant' is equivalent to the power of the actual experiment. Thousands of simulated experiments are needed to achieve reproducible power statistics. However, with today's personal computers, anyone can make very large series of simulations in one's own office.
The authors have written a set of programs intended for general use, which should enable any medical colleague to plan group sequential experiments. The programs may be run on personal computers using MS-DOS, and has been used to obtain the data presented below.
During group sequential experiments, statistical tests must be performed on levels of significance not found in ordinary tables. Programs to solve this difficulty are furnished on demand.
EXAMPLE: COMPARISON OF A CLASSICAL TWO-GROUP PLAN WITH EQUIVALENT, SEQUENTIAL PLANS.
The simplest of all group sequential plans would be a plan with only one interim test. How would if affect a fixed sample plan to be modified by performing a single interim test?
Imagine that we are planning an experiment to show the beneficial effect of a drug versus placebo. The effect is of the qualitative kind, with the response noted as a 'zero' or a 'positive'. The placebo is expected to yield about 30 per cent positives, the active drug about 60 per cent. The null hypothesis would be that the drug performs no better than the placebo, that is, we have equal 'true' proportions (30 per cent) of positives in the two groups.
Let us first design a classical fixed-size plan and test its performance (power) by simulation. Plan 1 in Table 2 is a fixed size plan, specified to the conventional Level of significance = 0.05, and a Power of 90 per cent. Conventional (and conservative) calculation led to a recommended sample size of 48.
The observed Power of 90.7 per cent is in good agreement with our expectations.
Let us compare Plan 1 in Table 2, with Plan 2. Plan 2 introduces an additional test when the experiment is half way through, on a Level of significance of 0.01. What is the effect on observed Power?
Simulations show that the total Power of Plan 2 is 91.0 per cent, practically the same as that of Plan 1. However, an additional benefit has been added: If the hypothesis of a 30 % difference is true, chances are 40.6 per cent that the interim test will reveal it, so that the experiment can be terminated half way through.
Curiosity drive us on to Plan 3, where the significance levels of the two tests have been switched. The effect of this is to diminish Power to 82.6 percent. But the chance of terminating the experiment early, if the hypothesis is true, has risen to 70.2 per cent.
Note that the total level of significance of Plan 2 and Plan 3 has not risen very much by the addition of an interim test.
Thus, we see that the addition of one interim test to an ordinary fixed group plan has led to tangible advantages, without any additional calculation, and without much sacrifice in terms of level of significance or Power. This device might be used routinely, and would in the long run lead to marked savings in suffering, time and money (5).
What happens if, instead of just one interim test, several interim tests are used? This may be done in many ways, the simplest seeming to be that interim tests are performed at regular intervals (in terms of observations), and that the same level of significance is used for all the tests of the plan, interim as well as final. Approximate levels of significance may be found from Table 1. Plans 4, 5 and 6 have been tabulated after testing several significance levels and sample sizes systematically by simulations, using the 'SEQ*' programs.
Plan 4 leads to a fall of total Power from 90.7 to 85.1 per cent. The chance of terminating the experiment half way through if the hypothesis is correct, is as high as 52.7.
Plan 5: What if we insist on keeping the Power at a minimum 90 per cent? A few trial runs show that to achieve this, maximum sample size must be set to 60. The number of interim tests is still 3. The chance of finishing early if the hypothesis is correct is still high, in fact there is a 35.9 per cent chance to stop as early as after 15 observations.
Plan 6, where we have five interim tests, instead of the three used in Plan 5, seems to lead to very much the same results with regard to Power as Plan 5. There is a bit more work to be done (in administration and testing), but also more opportunities of stopping early.
Comparison of Plan 5 and Plan 6 illustrates the general conclusion that the benefit of group sequential plans may be reaped using only a few interim tests.
DISCUSSION
We think that these examples demonstrate how medical researchers could master the planning of sequential experiments of the group sequential type. They also show the gains that may be achieved in this way. For the present examples, a qualitative (binomial) experiment has been used, but experiments using quantitative observations (measurements) may be planned in the same manner.
An average saving in study size of 1/3 compared to a fixed-size study is possible. The exact amount saved by a particular plan may be found by a few sets of simulations.
However, 'there are no free lunches'. To achieve this average saving while keeping up the power, maximal sample size must be set larger than the sample size of a comparable fixed size study. Exactly how much larger depends mainly on the number of interim tests, and when they are performed. The larger the number of interim tests, and the earlier in the study these are performed, the larger will maximal study size need to be set, in order to maintain adequate power. Again, in a given instance, one can find this out by a few sets of simulations.
It is in the nature of sequential plans that the final size of the study is unknown at the start. However, as long as a maximal size is stipulated, this uncertainty seems a fair price to pay for the potential gains in suffering, in time and in money.
CONCLUSION
Designing a sequential plan which is practical as well as safe seems comparatively easy when using computer simulation. Hopefully, the programs now available from the authors will encourage colleagues to attempt experiment plans of this type.
Table 1
NOMINAL LEVELS OF SIGNIFICANCE, EQUAL FOR INTERIM TESTS AND
FINAL TESTS, AND SUCH THAT THE REALISED TOTAL LEVEL OF
SIGNIFICANCE IS 0.050. *
+++++++++++++++++++++++++++++
Number of Nominal
interim level of
tests significance
-----------------------------
0 ** 0.0500 **
1 0.0296
2 0.0221
3 0.0183
4 0.0159
5 0.0142
6 0.0130
7 0.0120
8 0.0113
9 0.0107
+++++++++++++++++++++++++++++
* From (4), where more extensive tables are found. The values
may be used as conservative and very good approximations in
most cases. Precise values for a particular plan may be
found by simulation using the 'SEQ' programs.
** Fixed group trial.
Table 2
THE POWER OF ONE FIXED SIZE AND FIVE GROUP SEQUENTIAL
(R.S.T.) EXPERIMENT PLANS. DESIGN: TRIAL AIMED AT
DEMONSTRATING A HYPOTHETICAL 'TRUE' CONTRAST OF 30 PER CENT
VERSUS 60 PER CENT 'POSITIVES' (BINOMIAL TRIAL). EACH
ENTRY IS BASED ON 64000 SIMULATED EXPERIMENTS.
I = number of interim tests. N(max) = maximal sample size.
Percentage of tests 'significant' = Observed power.
+++++++++++++++++++++++++++++++++++++++++++++++++++
Sample Significance Total signifi- Percentage
size level cance level of tests
at test (nominal) (observed) 'significant'
---------------------------------------------------
PLAN 1: I = 0, N = 48
---------------------
48 0.05 0.050 90.7
PLAN 2: I = 1, N(max) = 48
--------------------------
24 0.01 40.6
48 0.05 0.053 91.0
PLAN 3: I = 1, N(max) = 48
--------------------------
24 0.05 70.2
48 0.01 0.052 82.6
PLAN 4: I = 3, N(max) = 48
--------------------------
12 0.0181* 27.4
24 " 52.7
36 " 75.6
48 " 0.050 85.1
PLAN 5: I = 3, N(max) = 60
--------------------------
15 0.0181* 35.9
30 " 63.2
45 " 84.0
60 " 0.050 91.8
PLAN 6: I = 5, N(max) = 60
--------------------------
10 0.0140* 24.3
20 " 47.9
30 " 62.7
40 " 77.2
50 " 86.4
60 " 0.050 91.3
+++++++++++++++++++++++++++++++++++++++++++++++++
* The Levels of significance for Plans 4 to 6 were found by
preliminary simulations, assuming that BOTH populations had
30 per cent 'positives', and starting with the values
0.0183 and 0.0142 found in Table 1.
REFERENCES
OTHER COMPUTER PROGRAMS
The authors do not know of other programs designed for the present purpose of enabling medical colleagues to design group sequential experiments. There are several programs written for statisticians, of which we have been recommended 'EaSt' (Early Stopping, from Cytel Software).
Your comments are welcome on: egilhl@online.no
This article was finished 14. January 1997, and was last updated on 2. December 1999.
Download Program (MEDSEQ.ZIP)
Home page Department
Home page University
Department of Public Health and Primary Health Care, last updated 16.04.07
Hogne.Sandvik@isf.uib.no