Post 8 — What if the assumptions are wrong?
Diagnostics can tell us that a model looks reasonable.
But they cannot tell us what would happen if the assumptions were violated.
That is where robustness checks become useful.
One simple example is unequal variance.
If two groups have substantially different variances, the classical equal-variance t-test is no longer the obvious choice.
Welch’s t-test provides an alternative by relaxing the equal-variance assumption.
Another approach is resampling.
For a contrast such as Method B − Method C, we can construct a bootstrap confidence interval by repeatedly resampling observations within the two methods and recalculating the contrast.
The question becomes intuitive:
How much would this estimated difference vary if the experiment were repeatedly resampled?
We can also use heteroscedasticity-consistent standard errors, such as HC3, to assess how sensitive inference is to non-constant variance.
And we can deliberately introduce an extreme outlier.
That last exercise is particularly revealing.
In the example dataset, an artificial outlier changes the estimated mean for Method B substantially.
Yet the broad conclusion that the methods differ remains.
This gives us an important distinction:
a conclusion can be robust even when an estimate is not.
The global statement “methods differ” may survive.
The exact estimated mean or contrast may not.
That is why I increasingly think of robustness as a form of sensitivity analysis.
We are not looking for a second statistical method that produces the “correct” answer.
We are asking:
Which parts of our conclusion depend strongly on the assumptions we chose?
That is much more informative than simply reporting another p-value.
The implementation details and reproducible analysis for this step are available in the GitHub repository at https://github.com/andreabz/analytical-anova/ and on the project page at https://andreabz.github.io/analytical-anova/
