Hypothesis testing
1.P- value
p value is measure the probability of the outcome which are more extreme than what you have observed under null hypotheis.
2. Power analysis
drow the probility of rejection criteria verying from different parameter domain.
Null hypothesis domain:
Ω
n
u
l
l
=
{
θ
n
u
l
l
}
Ω
a
l
t
=
{
θ
a
l
t
}
\Omega_{null} =\{ \theta_{null}\} \;\;\;\;\; \Omega_{alt} = \{ \theta_{alt}\}
Ωnull={θnull}Ωalt={θalt}
we should desige a test
δ
\delta
δ
such that the we bond our false positive rate smaller than a significant level (5%).
m
a
x
θ
n
u
l
l
P
(
r
e
j
e
c
t
i
o
n
)
<
=
5
%
\underset{\theta_{null}}{max} P(rejection) <= 5\%
θnullmaxP(rejection)<=5%
When we lower the FPR, inevetibly we lower TPR rate as well since the function is continuous.
Think about the boundary of \theta_{null}, it is positive case but the rejection probability is ~ 5%, which the TRP rate is 5% as well. So we want to design our power to change rapidly about the boundary, (green test is better than black test). This is achieved by lower the variance or increasing the sample size.
3.One side vs two side
-
one side:
n u l l : μ > μ 0 ; v s a l t : μ ≤ μ 0 null: \mu \gt \mu_0 ;\;\;vs\;\;alt: \mu \leq \mu_0 null:μ>μ0;vsalt:μ≤μ0 -
two side (the plot above is showing a two side test example)
n u l l : μ = μ 0 ; v s a l t : μ ≠ μ 0 null: \mu = \mu_0 ;\;\;vs\;\;alt: \mu \neq\mu_0 null:μ=μ0;vsalt:μ=μ0