Post 2 — The p-value does not tell you which comparison matters
Once ANOVA tells us that the four methods do not all have the same mean, the natural question is:
Which differences actually matter?
This sounds obvious, but it is where the analysis can easily become mechanical.
R gives us coefficients relative to a reference level. That is useful, but the choice of baseline is a parameterisation choice, not a scientific decision.
Suppose the interesting question is whether Method B differs from Method C.
There is nothing special about the fact that neither is the baseline.
We can write the scientific question directly as a contrast:
B − C.
More generally, a contrast is a linear combination of means whose coefficients sum to zero.
That simple definition is surprisingly powerful.
It allows us to encode questions such as:
Does Method B differ from Method C?
Is the Reference method different from the average of the three alternatives?
Is the average performance of two methods different from another method?
The important point is that these comparisons should come from the scientific problem, not from whatever happens to appear convenient in the coefficient table.
There is also a second issue: multiple comparisons.
If we start testing every pair we can think of, the probability of obtaining at least one apparently significant result simply because we tested many hypotheses increases.
That is why procedures such as Tukey’s HSD exist.
But multiplicity correction should not be confused with scientific relevance.
A very small difference can be statistically significant.
A large difference can be scientifically important even when the experiment is too imprecise to establish statistical significance.
In analytical chemistry, I think the most useful habit is therefore to ask two separate questions:
Is there evidence that the difference is non-zero?
and
Is the magnitude of that difference relevant to the analytical application?
Those questions are related.
They are not the same question.
The implementation details and reproducible analysis for this step are available in the GitHub repository at https://github.com/andreabz/analytical-anova/ and on the project page at https://andreabz.github.io/analytical-anova/
