README.TXT
----------

Code to accompany "Statistical Decision Properties of Imprecise Trials
Assessing COVID-19 Drugs"
by Charles F. Manski and Aleksey Tetenov
June 2020

Computations were performed in MATLAB R2018b.
Most programs require MATLAB Statistics and Machine Learning Toolbox

Contact aleksey.tetenov@unige.ch if you have questions about this code.



Code for two-arm trials with binary outcomes
--------------------------------------------------------

nearopt_twoarm_specific.m
	Evaluates expected loss of the empirical success rule and 
	the hypothesis test rule in two-arm trials with binary outcomes
    for specific hypothetical average outcome values.
	Used to compute the results in Table 1.

Table1.m
	Computes the results appearing in Table 1.

nearopt_twoarm.m
	Evaluates near-optimality of the empirical success rule and
	the hypothesis test rule in two-arm trials with binary outcomes.
	Grid search over a specified range of possible average outcome
	values.
	Used to compute the results in Table 2.

Table2.m
	Computes the results appearing in Table 2.



Code for multi-arm trials with binary outcomes
----------------------------------------------------------

dunnett_unequal.m
	Computes Dunnett's test two-sided critical value.
	Requires MATLAB Optimization Toolbox.

nearopt_multiarm_specific.m
	Evaluates expected loss of the empirical success rule and 
	Dunnett's test rule in multi-arm trials with binary outcomes
    for specific hypothetical average outcome values.
	Used to compute the results in Table 3.

Table3.m
	Computes the results appearing in Table 3.

nearopt_multiarm.m
	Evaluates near-optimality of the empirical success rule and
	Dunnett's test rule in multi-arm trials with binary outcomes.
	Grid search over a specified range of possible average outcome values.
	This function is suitable for any combination of per-arm sample sizes,
	but could be very time-consuming.
	Requires MATLAB Optimization Toolbox for computing the critical value
	of Dunnett's test.



Calculations for Table 4 proceed in three steps:
------------------------------------------------
grid.m
	Step 1/3
	This script performs a global grid search in increments of 0.02
	to estimate near-optimality of the empirical success rule and 
	of Dunnett's test rule in trials with 5 treatment arms.
	It is assumed that arms 2-5 have the same sample size.
	Expected loss is approximated by simulating 100,000 trial outcomes
	for each set of parameter values.
	Requires MATLAB Optimization Toolbox for computing the critical value
	of Dunnett's test.

finegrid.m
	Step 2/3
	This script performs a finer grid search to estimate near-optimality 
	of the empirical success rule and of Dunnett's test rule in trials
	with 5 treatment arms. It is assumed that arms 2-5 have the same
	sample size.
	The results of a global grid search in step 1 suggest that the expected
	loss for the empirical success rule is largest when the parameters (average
	success rates of treatments) have the form
		[a b b b b], a > b
	and for Dunnett's test rule the largest expected loss occurs
	for parameters of the form
		[a b c c c], b > a, b > c
	The script constructs two grids of parameter values in increments of 0.01
	having these forms and searches over them.
	Expected loss is approximated by simulating 1,000,000 trial outcomes
	for each set of parameter values.
	Requires MATLAB Optimization Toolbox for computing the critical value
	of Dunnett's test.

verify.m
	Step 3/3
	Takes 10 parameter combinations with the largest estimated expected loss
	for each decision rule found in step 2 (finer grid search) and
	re-computes expected loss for these parameters at higher precision,
	by simulating 100,000,000 trial outcomes.
	This is done to verify that our results are not affected by bias
	from approximating the sampling distribution by simulation.
	The results appearing in Table 4 of the paper are based on this output.
