Statistical Decision-Making should control the Incorrect Decision Rate

Music City Center 

Statistics is truly a science of facilitating data informed decisions – to control and minimize the Incorrect Decision rate (such as incorrectly targeting a patient subgroup). I would like to share some intriguing observations regarding some commonly used procedures.
One interesting finding is the log-rank test does not control the Familywise Type I error rate strongly, when the log-rank test is viewed as a test combining N individual 2x2 contingency table tests (where N is the number of unique event times). The danger of making an incorrect decision upon the rejection of a log-rank test is classic: interpreting the cause of the rejection post hoc. It is the reason why usually strong control of the Familywise Type I error rate is required. Thus, in survival analysis, to control the incorrect decision rate, decision-making should incorporate a logic respecting efficacy measure such as ratio of survival times as well.
I general, an inflated Incorrect Decision rate when the Type I error rate is seemingly controlled is due to "rejecting for wrong reasons" not being counted. I will give an illustrative example that, in 2-sided testing involving multiple doses and endpoints, there is a danger of incorrectly labeling a compound as having efficacy in a secondary endpoint when there is not, if directional error rate is not counted as a Type I error. This could happen with alternative primary endpoints. Efficacy is directional, so in 2-sided testing, directional error should be counted. Confidence set methods are recommended, because they automatically control the directional error rate.
Statisticians are guardians of science - where we shine the most.