Statistical and Trial Design Considerations in Central Nervous System Prophylaxis Studies

End Points for CNS Disease 

The primary end points selected to assess clinical outcome should balance the competing goals of patient benefit and scientific rigor sufficient to satisfy the demands of institutional review boards and regulatory agencies.20

Time to Neurologic Progression (clinical) 

Time to neurologic progression, which censors patients at the time of death, and progression-free survival (PFS), which includes both progression and death as events, represent end points that attempt to combine time and quality of survival in one measure (Table 1). Traditionally, these outcome measures do not account for treatment-related neurotoxicities, which would be necessary to evaluate the net benefit of treatment. Meticulous prespecification of the clinical definition of neurologic progression is essential. Once clearly defined, the assessment of neurologic progression must be blinded, since significant bias is likely when the individual making outcome determinations is not masked to the treatment assignment. Assessments performed by a designated outside physician (someone other than the treating physician) may help overcome this potential problem. Data review and end point determination by an independent expert committee may be another alternative.21, 22

Table 1. Traditional End Points for Neurooncology Clinical Trials

	End Point	Confounders and Concerns
	Tumor response	Lack of correlation with QoL Test imprecision
	Survival	Variation in subsequent care Death from non–central nervous system cancer Ignores quality of survival
	Cause of death	Variation in subsequent care Lack of reliable criteria Poor correlation with other end points
	Surrogate markers	Restricted availability Lack of correlation with QoL
	Time to tumor progression (radiographic or cytologic)	Lack of correlation with QoL Test imprecision
	Time to neurologic progression	Usually unblinded Test imprecision Cumbersome

Abbreviation: QoL, quality of life.

The frequency of assessment of neurologic progression, and the interval between repeated tests, are critical as well. Assessment frequency is in large part determined by the size of the expected effect; if treatment is expected to delay progression by only a few weeks, the assessment must be more frequent then if a longer delay is expected. This desire for frequent testing is balanced against the increased cost and feasibility of testing. Both increased frequency of testing and increased sample size increase the power of the study. Generally, the more frequently neurologic progression is assessed, the fewer the number of patients required to complete the trial. To avoid the introduction of bias, assessment frequency must be equal between comparison groups.

QoL and Neurocognitive End Points 

Direct measures of clinical benefit include QoL and neurocognitive end points. These measurements allow the clinician to evaluate the trade-off between clinical benefit and treatment-related toxicity (both neurologic and non-neurologic) and assess the relative impact of treatment on the patient's QoL.

QoL 

QoL end points include strategies such as Quality-adjusted Time Without Symptoms of disease or Toxicity of treatment (Q-TWiST)23 and the Independent Living Score (ILS).24 Q-TWiST provides a single measure that integrates quality and quantity of life. This is a formal mathematical technique that allows the clinician to assess the amount of time a patient experiences treatment-related toxicity, the proportion of the total survival time during which a patient is free of symptoms related to either treatment or disease progression, and the amount of time during which symptoms are present because of disease progression.25 The rationale for the ILS is the understanding that increasing impairment associated with cancer progression decreases independent functioning. Therefore, a simple way to measure QoL may be to quantify the amount of time a patient remains independent in his or her activities. While reductionist, and a simplification of the concept of QoL, this measurement approach makes the reasonable assumption that one critical global variable (independence) measured over time will reflect other aspects of a patient's QoL.24

Neurocognitive Function 

Brain tumors may profoundly affect cognitive function (eg, memory, decision-making), as may the treatment. Neurocognitive performance may thus reflect the net benefit of treatment over toxicity. Many instruments measure neurocognitive function (Table 2).26, 27, 28, 29, 30, 31, 32, 33 However, given the multifactorial causes of cognitive deficits in patients with CNS metastases, accurate assessment may present a considerable challenge.26, 34 No single, simple screening tool reliably identifies the degree and course of impairment and also defines the contributions of multiple etiologies. Some tests measure discrete processes such as expressive language function, while others measure more distributed, nonspecific, functions throughout the brain.34 Because neurocognitive deficits may limit a patient's ability to complete the tests, the reason for non-completion also must be captured, thereby increasing the power of this end point. The utility of most tests depends on the psychometric properties of the measure, the nature of the populations and treatment under study, and the effect on neurocognitive function of those treatments and of the disease itself.

Table 2. Instruments Available to Assess Neurocognitive Function

	Test	Characteristic Measured	Time to Administer (min)
	Hopkins Verbal Learning Test27	Verbal memory	5
	Trail Making Test Part A28	Visual-motor speed	2
	Trail Making Test Part B28	Executive function	5
	Controlled Oral Word Association31	Verbal fluency	5
	Digit Span Test32	Attention and concentration	10
	Folstein MMSE33	Global cognitive function	10
	Grooved Pegboard29 (non-dominant and dominant hands)	Motor dexterity and speed	2-3
	Categoric Word Fluency30	Executive function	5

Abbreviation: MMSE, Mini-Mental State Examination.

Few studies have prospectively followed neurocognitive function among patients treated in randomized clinical trials who are suffering from CNS metastases.35, 36 Corn et al used the Folstein Mini-Mental State Examination (MMSE), a simple and widely used tool to assess dementia.35 Mehta et al used a sophisticated array of neurocognitive tests, including the Hopkins Verbal Learning tests, Controlled Oral Word Association, Grooved Pegboard Test, and TrailMaking tests A and B.36 Performance on neurocognitive testing also depends, in part, on the patient's compliance with treatment and testing. Compliance observed in the Mehta study was high (94%-96%), and almost all patients in the Corn study completed at least one MMSE. These studies support the feasibility of formally evaluating neurocognitive outcome among patients treated for brain metastases. Although many of these tests are relatively simple to administer, a high degree of professional expertise is required for their accurate interpretation.26, 34

Trial Design Considerations 

An effective trial design calls for adequate numbers of prespecified primary end point events. An adequate number of events can be achieved by enrolling a large enough number of patients, enrolling a smaller number of selected patients more likely to achieve the specified event, liberalizing the definition of the primary outcome measure, or by extending the duration of follow-up. Each of these strategies has its own benefits and drawbacks. Accruing large numbers of patients and prolonging the length of the trial increase expenses and make administration of the trial more complex. Selecting an at-risk population more likely to reach a study end point is more economical but may slow down enrollment and limit the ability to generalize the results. One important advantage of using an at-risk population in the context of a prophylaxis trial is a reduction in the number of patients who receive treatment unnecessarily, ie, patients who would not have suffered the outcome of interest even without the study intervention. Another way of looking at this trade-off are the concepts of “number needed to treat” (NNT) and “number needed to harm” (NNH), concise expressions of the number of patients who would need to undergo the proposed intervention in order for one patient to benefit from or be harmed by the intervention. An example of these concepts as they apply to a trial of CSF prophylaxis in patients with NHL may be found in the article by Herrlinger et al in this supplement.

In a large-scale retrospective study, Hollender et al sought to determine the incidence and risk factors for CNS relapse in patients with NHL.4 A primary objective of this study was to establish risk assessment guidelines to help clinicians identify patient subpopulations that might be ideal targets for CNS prophylaxis. The probability of CNS recurrence within 5 years was calculated as a function of the number of risk factors present. CNS prophylaxis was recommended for patients with high-grade NHL and at least four risk factors (see Table 5 in the article by Herrlinger et al in this supplement).

Risk models such as this may indeed help clinicians identify high-risk patients and decide whether to implement CNS prophylactic therapy. From a clinical trials perspective, the advantage of using such a model to guide patient enrollment is the greater power to detect a treatment effect in a smaller sample size. As noted, however, the ability to generalize study results and a reduced pool of study-eligible patients (and hence slower patient accrual) are potential trade-offs.

Randomizing small patient numbers may lead to other methodologic challenges. For example, the desire to stratify patients (by study site or other variables) using the usual blocked lists for randomization may become too predictable. One possible way to avoid this predictability is to use a centralized adaptive randomization that achieves balance across stratification variables and across sites without using the usual prespecified blocked lists. With this approach, each subsequent patient randomization depends on the characteristics of the preceding randomization decision, and in this way, allocation concealment is preserved.

Because the power of a study (for time to event end points) is driven by the number of events, not the entire sample size, the usual strategy for powering a trial assumes a targeted event rate and calculates the number of patients to achieve the desired number of events. There are four basic methods for preserving power in a given trial: (1) increase the number of observed events either by extending the follow-up period, increasing the frequency of testing, or redefining the end point; (2) increase the hypothesized effect size (a strategy that is only acceptable if the investigators believe the revised effect size is valid); (3) increase alpha; or (4) use a one- versus two-tailed test. (In some contexts, a two-tailed test is not necessary; for example, the traditional Simon Two-Stage Design is a one-tailed test.)

Commonly, enrollment is based on a fixed sample size that was generated by assuming a targeted event rate. Because the power of a trial is critically dependent on this potentially uncertain rate, a safer approach may be to base enrollment on the estimated number of events, not on sample size (ie, continue enrollment until a predefined number of events is achieved). This approach maintains statistical power but comes with the trade-off of an uncertain sample size that cannot be specified at study outset. Furthermore, this approach requires a relatively short time between the occurrence of the event and collection of pertinent data by the study personnel.

Proposed GBM Phase III Study 

The phase III GBM trial proposed elsewhere in this supplement considers two possible end points: survival (4-year accrual, 18-month follow-up) and PFS (4-year accrual, 12-month follow-up) (see the article by Glantz et al in this supplement). This trial assumes a median survival of 14.6 versus 19.5 months and a median PFS of 6.9 versus 9.2 months in the standard versus investigational treatment arms, both with a hazard ratio (HR) of 0.75. Survival and PFS estimates for the standard treatment arm are based on published results from a well-designed phase III trial.43 Survival and PFS targets for the investigational arm are based on the investigators' judgment about what would constitute a clinically meaningful response advantage. Table 3 provides the number of patients required to detect these differences for various values of alpha and power levels. Of note, although the medians differ for the two end points (survival v PFS), the proposed treatment effect (HR) is the same. The follow-up time and the number of patients needed to achieve this effect size are smaller when PFS is chosen as the end point, because the PFS end point results in an earlier and a greater number of events.

Table 3. Glioblastoma Phase III Study Design (survival end point, 4-year accrual, 18-month follow-up; progression-free survival end point, 4-year accrual, 12-month follow-up)⁎

	14.6 Versus 19.5 Months Median Survival (HR, 0.75)
		Sample Size Per Arm
		85% Power	90% Power
	Alpha = 0.05	275	321
	Alpha = 0.10	220	262

	6.9 Versus 9.2 Months Median PFS (HR, 0.75)
		Sample Size Per Arm
		85% Power	90% Power
	Alpha = 0.05	236	276
	Alpha = 0.10	189	225
	Alpha = 0.20	142	173

Abbreviations: PFS, progression-free survival; HR, hazard ratio.

⁎ Assumes log-rank tests for all comparisons.

Alternatively, dramatically smaller sample sizes and time required for patient accrual could be achieved with a single-arm trial using a PFS end point. However, the lack of blinding in these types of single-arm trials has the potential to introduce substantial observer bias, and the lack of randomization requires faith that the historical control population is very comparable to the study population and that the historical outcome data are reliable.

Proposed NHL Prophylaxis Study 

The phase III NHL prophylaxis study proposed by Herrlinger et al elsewhere in this supplement also would consider two end points: time to progression (2 years accrual, 2 or 3 years follow-up) and PFS (2 years accrual, 2 or 3 years follow-up) (Table 4). Required risk factors for study eligibility would be elevated lactate dehydrogenase and either involvement of more than one extranodal site or a specific site of disease associated with a high risk of CNS relapse. The sample size calculations would assume a two-sided alpha of 0.05, both risk factors present, and a time to progression event rate of 10% versus 4% and PFS of 60% versus 68.2% at 2 years (Table 4). The authors also would consider the benefit-to-risk trade-off by comparing the NNT to prevent one case of CNS relapse with the NNH. Assuming a 10% CNS relapse rate, the NNT would be 13.6 and the NNH 3.5, with an odds ratio in favor of prophylaxis of 3.9 (95% confidence interval, 1.4-11.0). Using the risk factors and assumptions outlined in their proposal, these ranges fall within what is usually considered acceptable in other disease models.

Table 4. Non-Hodgkin Lymphoma Phase III Prophylaxis Study Design (time to progression end point, 2-year accrual, alpha = 0.05; progression-free survival end point, 2-year accrual, alpha = 0.05)⁎

	TTP: Two or More Risk Factors: 2-Year Event Rate 10% v 4%
		Sample Size Per Arm
		85% Power	90% Power
	2-Year follow-up	236	276
	3-Year follow-up	180	210

	PFS: 60% v 68.2% Event-Free at 2 Years (HR, 0.75)
		Sample Size Per Arm
		85% Power	90% Power
	2-Year follow-up	460	538
	3-Year follow-up	379	443

Abbreviations: TTP, time to progression; PFS, profression-free survival; HR, hazard ratio.

⁎ Assumes log rank tests for all comparisons.

Once again, the length of time patients are followed (2 v 3 years) has a significant impact on the number of patients required to complete the study, since a longer follow-up period translates into a higher number of events. The desired power (85% v 90%) also influences study size and reliability. A lower power conveys a higher probability of a type II error. For example, an 80% power incurs a 20% chance of a type II error. In earlier drug development settings (phase II), adequate control of the type II (false-negative) error may be extremely important, because a false-negative result at this stage could prematurely terminate the development of a promising agent. In late stages of development, the burden shifts to avoiding a false-positive result (a type I error) to avoid the widespread use of a drug that is ineffective.44

Herrlinger et al also offer a randomized phase II “pick the winner” trial design .45 In this design, the two treatment arms would each be compared independently to a prespecified historical control but not to each other. This design may be practically superior, since study size would be unreasonably large if a phase III trial design was used, and successful accrual to a phase III trial might be very difficult to achieve in this patient population. Table 5 provides sample size estimates for various response assumptions and power choices for this randomized phase II “pick the winner” design, a randomized phase III design with the end point of relapse rate, and a single-arm phase II design with an historical control comparison. These calculations illustrate the substantial variation in sample size dictated by choice of study design and by other factors such as estimates of effect size and choice of power.

Table 5. Sample Size Calculations for a Proposed non-Hodgkin Lymphoma Prophylaxis Study Using Various Designs (primary outcome variable is defined as relapse in the central nervous system)

	Randomized Controlled Phase III Design (end point: relapse rate)⁎
	Relapse Rates		Sample Size (per arm)
	Treatment 1 (L-AraC)	Treatment 2 (MTX)	80% Power	90% Power
	0.01	0.07	180	225
	0.02	0.06	400	520
	0.03	0.05	1,600	2,100

	Single-Arm Phase II Model†
	Relapse Rates		Sample Size
	Treatment (L-AraC or MTX)	Historical High-Risk Population	80% Power	90% Power
	0.01	0.15	35	35
	0.02	0.15	35	46
	0.03	0.15	46	56
	0.05	0.15	75	102

	Randomized Phase II “Pick the Winner” Design‡
	Relapse Rates		Sample Size (per arm)
	Treatment (L-AraC or MTX)	Historical High-Risk Population	80% Power	90% Power
	0.01	0.15	35	35
	0.02	0.15	35	46
	0.03	0.15	46	56
	0.05	0.15	75	102
	0.06	0.15	102	127
	0.07	0.15	127	160

NOTE. All calculations assume a two-sided alpha of 0.05. Abbreviations: L-AraC, liposomal cytarabine; MTX, methotrexate; NHL, non-Hodgkin lymphoma.

⁎ Experimental treatment reduces the relapse rate to 1% to 3%, while the relapse rate varied between 5% and 7% in patients treated with MTX. Test: Fisher's exact test. † Test: one-sample exact test of a binomial proportion. ‡ Test: Two separate one-sample exact tests of a binomial proportion. Power is for each test independently.

Conclusion 

Definitive evidence for the effectiveness of CNS prophylaxis is lacking except in small cell lung cancer,46, 47 acute leukemia, and very aggressive lymphomas (ie, Burkitt lymphoma and lymphoblastic lymphoma). In other patient cohorts at high risk for CNS relapse, prophylactic therapy may represent an important treatment option; however, the patient pool for clinical trial participation is relatively small. A small number of patients magnifies the challenges of effective trial design. Researchers involved in trial design should be particularly mindful of the following issues that could be detrimental to a successful study. Do not manipulate the effect size simply to achieve a manageable estimate of sample size. This will likely result in too few patients to arrive at a meaningful conclusion at the end of the study. Consider centralized randomization to avoid predictable treatment allocation that could bias trial results. Consider both the risks (slower accrual, possibly decreased generalizability) and the benefits (smaller study size, shorter duration, fewer patients exposed to the intervention in prophylactic trials) of limiting the study to an at-risk population. Avoiding type II error is perhaps the most important goal of early-phase trials; therefore, consider a one-tailed test or a two-stage test that does not have higher-than-desired probability of early termination. Do not modify an end point simply to increase the number of events (or power) at the expense of using the most clinically relevant or treatment-responsive end point. Blinded outcome assessment, randomization, and stratification for prognostically important variables are also critically important considerations. More research directed at optimal trial design and outcome measures is needed for CNS prophylaxis trials.