Hypothesis Testing: Part 3 ANOVA
In the last part (Part_2)of this series we learn about the T-test, and Chi-square Goodness of Fit Test. Now in this part, we are going to learn about ANOVA.
ANOVA (Analysis of Variance) is used to check if at least one of two or more groups have statistically different means. Now, the question arises — why do we need another test for checking the difference of means between independent groups? Why can we not use multiple t-tests to check for the difference in means?
The answer is simple. Multiple t-tests will have a compound effect on the error rate of the result. Performing t-test thrice will give an error rate of ~15% which is too high, whereas ANOVA keeps it at 5% for a 95% confidence interval.
To perform an ANOVA, you must have a continuous response variable and at least one categorical factor with two or more levels. ANOVA requires data from approximately normally distributed populations with equal variances between factor levels. However, ANOVA procedures work quite well even if the normality assumption has been violated unless one or more of the distributions are highly skewed or if the variances are quite different.
ANOVA is measured using a statistic known as F-Ratio. It is defined as the ratio of Mean Square (between groups) to the Mean Square (within-group).
Mean Square (between groups) = Sum of Squares (between groups) / degree of freedom (between groups)
Mean Square (within-group) = Sum of Squares (within-group) / degree of freedom (within-group)
Steps to perform ANOVA
- Hypothesis Generation
a. Null Hypothesis: Means of all the groups are the same
b. Alternate Hypothesis: Mean of at least one group is different
2. Calculate within group and between groups variability
3. Calculate F-Ratio
4. Calculate probability using F-table
5. Reject/fail to Reject the Null Hypothesis
There are various other forms of ANOVA too like Two-way ANOVA, MANOVA, ANCOVA, etc. but One-Way ANOVA suffices the requirements of this series.
Practical applications of ANOVA in modeling are:
1. Identifying whether a categorical variable is relevant to a continuous variable.
2. Identifying whether a treatment was effective to the model or not.
Practical Example
Question:- A researcher wishes to see if there is a difference in the fuel economy for city driving for three different types of automobiles: small automobiles, sedans, and luxury automobiles. He randomly samples four small automobiles, five sedans, and three luxury automobiles. The miles per gallon for each is shown. At a = 0.05, test the claim that there is no difference among the means.
Solution:- I solve this question from scratch using Python.
Follow the Python Notebook over here!
Step1: State the hypotheses and identify the claim.
H0: m1 =m2 =m3 (claim)
H1: At least one mean is different from the others.
First, we need to get the data into Python:
The data now looks as follows:
Step 2: Find the critical value.
N =12 , k =3
d.f.N. =k -1 =3 -1 =2
d.f.D. =N -k =12 -3 =9
The critical value from Table H in Appendix A with a =0.05 is 4.26.
Step 3: Compute the test value.
a. Find the mean and variance for each sample. (Use the formulas in Chapter 3.)
For the small cars: mean = 37.25, variance = 20.91
For the sedans: mean = 35.4, variance = 37.3
For luxury cars: mean = 26, variance = 7
b. Find the Grand Mean
c. Find the between-group variance
d. Find the within-group variance
e. Calculate F-Test
Step 4: Make the decision. The test value 4.83 4.26, so the decision is to reject the null hypothesis.
You can use the F-calculator here.
Note: ANOVA only lets us know the means for different groups are the same or not. It doesn’t help us identify which means are different. To know which group mean is different, we can use another test known as Least Significant Difference Test.
