Post 7 — A model can be significant and still be wrong
After fitting several linear models, it is tempting to stop once the p-values look convincing.
That is exactly when diagnostics should begin.
A classical linear model relies on assumptions about the residuals: independence, approximate normality and reasonably constant variance.
These assumptions are not ceremonial requirements.
They determine whether the usual standard errors and F-tests behave as expected.
For the interaction model in this project, the diagnostic plots are reassuring.
Residuals are reasonably scattered around zero.
There is no strong funnel shape.
The Q-Q plot is reasonably close to the reference line, with only modest tail deviations.
Cook’s distance does not reveal highly influential observations.
Formal tests for normality and heteroscedasticity are also broadly consistent with the visual assessment.
But I deliberately do not want to present this as:
“Shapiro-Wilk is non-significant, therefore the model is valid.”
That is a poor diagnostic strategy.
Formal tests depend strongly on sample size.
With enough observations, tiny deviations can become statistically significant.
With few observations, substantial deviations can remain undetected.
The plots tell us something different: whether the deviation appears large enough or structured enough to threaten the model.
There is also a deeper point.
Independence is primarily a property of the experimental design, not something a residual plot can prove.
You cannot diagnose away pseudoreplication.
You have to understand how the observations were generated.
Diagnostics therefore do not answer:
“Is the model true?”
They answer a more useful question:
“Are there features of the data that make the model’s assumptions implausible enough to affect the conclusions?”
That is a much more defensible way to think about model checking.
The implementation details and reproducible analysis for this step are available in the GitHub repository at https://github.com/andreabz/analytical-anova/ and on the project page at https://andreabz.github.io/analytical-anova/
