Confidence level and significance level

Answer 1

I have always been careless about the use of the terms "significance level" and "confidence level", in the sense of whether I say I am using a 5% significance level or a 5% confidence level in a statistical test. I would use either one in conversation to mean that if the test were repeated 100 times, my best estimate would be that the test would wrongly reject the null hypothesis 5 times even if the null hypothesis were true. (On the other hand, a 95% confidence interval would be one which we'd expect to contain the true level with probability .95.) I see, though, that web definitions always would have me say that I reject the null at the 5% significance level or with a 95% confidence level. Dismayed, I tried looking up economics articles to see if my usage was entirely idiosyncratic. I found that I was half wrong. Searching over the American Economic Review for 1980-2003 for "5-percent confidence level" and similar terms, I found: 2 cases of 95-percent significance level

27 cases of 5% significance level 4 cases of 10% confidence level

6 cases of 90% confidence level Thus, the web definition is what economists use about 97% of the time for significance level, and about 60% of the time for confidence level. Moreover, most economists use "significance level" for tests, not "confidence level".